Race condition between download of file and immediate use within same task in 5.x?

[chadlwilson]

After upgrading to 5.x the GoCD extension task here no longer works properly.

The extension task tries to automate downloading of a checksum file which it then compares against the binary; wrapping it into one custom task.

https://github.com/gocd/gocd/blob/77e7c7e32d03a175e145708037ba479aa430a8cf/buildSrc/src/main/groovy/com/thoughtworks/go/build/DownloadFile.groovy#L58-L70

What seems to happen is that the actual file is not available immediately after super.download() completes, so we get

Caused by: java.io.FileNotFoundException: /Users/chad/.gradle/download-cache/e1a5e9cf067f65361abbdb33096b3eb7/OpenJDK15U-jre_x64_mac_hotspot_15.0.2_7.tar.gz (No such file or directory)
        at de.undercouch.gradle.tasks.download.VerifyAction.execute(VerifyAction.java:62)
        at de.undercouch.gradle.tasks.download.VerifyAction$execute.call(Unknown Source)
        at com.thoughtworks.go.build.DownloadFile.download(DownloadFile.groovy:69)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)

Re-running the same task immediately afterwards will succeed, so it seems when download() returns, it is still doing something in the background that eventually does complete, but is not complete by the time VerifyAction.execute() is trying to read the destination file to calculate its checksum.

Is there something I am missing, and a way to make such an extension task work correctly with v5?

[michel-kraemer]

Download.execute() uses Gradle's Worker API to execute the task asynchronously now. Try to inherit from DownloadAction instead. It has an execute() method that returns a CompletableFuture, for which you can wait.

Please let me know if this helps or if there is anything else I can do.

[michel-kraemer]

I'm curious, why do you extend Download at all? Why don't you just define a task that depends on a Verify task that depends on a Download task? This way you could use Gradle's native task dependency mechanism.

[chadlwilson]

I'm not sure, as this was written a long time ago, but GoCD is a big many-project Gradle build and there are quite a few different places we download files when assembling installers. So I imagine the intent is to DRY somehow by decorating the Download task so the same dependent tasks don't have to be defined everywhere.

But there is no doubt different ways to achieve this and write a composite task if one knows ones Gradle inside out 😅

[michel-kraemer]

Ah OK. I see what you are trying to do. How about this one?

class DownloadFile extends Download {
  @Input
  @Optional
  Object checksum

  @Internal
  String expectedChecksum

  @Internal
  boolean shouldDownload

  DownloadFile() {
    doLast {
      if (shouldDownload) {
        project.logger.info("Verifying checksum of ${dest}")  
        if (expectedChecksum != null) {
          def action = new VerifyAction(project)
          action.checksum(expectedChecksum)
          action.algorithm('SHA-256')
          action.src(dest)
          action.execute()
        }
      }
    }
  }

  @TaskAction
  @Override
  void download() {
    if (checksum instanceof Callable) {
      expectedChecksum = checksum.call()
    } else if (checksum instanceof String) {
      expectedChecksum = checksum.trim()
    }

    shouldDownload = true

    if (dest.exists()) {
      def actualChecksum = dest.withInputStream { is -> DigestUtils.sha256Hex(is) }
      if (expectedChecksum != null) {
        project.logger.info("Verifying checksum. Actual: ${actualChecksum}, expected: ${expectedChecksum}.")
        shouldDownload = actualChecksum != expectedChecksum
      } else {
        shouldDownload = false
      }
    }

    if (shouldDownload) {
      project.logger.info("Attempting download of ${src} into ${dest}")
      super.download()
    } else {
      project.logger.info("Skipping download of ${src}")
    }
  }
}

I just tested it and it seems to do what you want.

[chadlwilson]

Oh wow, thanks a lot - that seems to work perfectly fine. Forgot about the ability to doLast inside constructor :-)

[michel-kraemer]

@chadlwilson My pleasure. Just a quick question: I'm currently thinking about adding a "Who's using" section to the README with logos of users of gradle-download-task. Would you mind if I added GoCD's logo or the Thoughtworks logo there? I would of course also link to your website or repository. Let me know if that's OK - you can also send me a private email. Thanks!

[chadlwilson]

@chadlwilson My pleasure. Just a quick question: I'm currently thinking about adding a "Who's using" section to the README with logos of users of gradle-download-task. Would you mind if I added GoCD's logo or the Thoughtworks logo there? I would of course also link to your website or repository. Let me know if that's OK - you can also send me a private email. Thanks!

Personally I don't see a problem with you adding a small GoCD logo/link to the Github repo - it's all open-source so by definition not a secret, and not dissimilar to the "Used By" stuff that appears on some repos through some Github dependency magic (e.g commons-io). If Github ever properly understands Gradle dependency trees, perhaps it'll even happen automagically :-) I'm not a trademark lawyer though, so will check with the other folks.

As for a Thoughtworks logo - that's a different thing. While I still currently work at Thoughtworks, I help to maintain GoCD only in my personal capacity so have no power to endorse or imply endorsement of anything. :-) Also, while GoCD was originally created and sponsored (until end of 2020) by Thoughtworks, it's now a fully open-source/community project, so nothing the contributors do is "in the name of Thoughtworks". We probably need to do a bit better making that clearer though, as there is still Thoughtworks branding (and it's outdated branding too!) in a number of places which makes this a bit confusing.

[michel-kraemer]

Cool! No worries about the Thoughtworks logo, I just wanted to ask if either of the two is possible. I've created a pinned issue to track who's using the plugin (#192). It would be awesome if you could check with your folks and then post the logo there! Thank you so much!!