Using Cue to replace depends_on in YAML.

[jer-k]

I'm trying to wrap my head around how I can use Cue to replace a YAML structure that has depends_on, such as docker-compose.yml.

Something simple might be a Rails API that depends on a Postgres database. Here is an example compose file I have.

version: '3'
services:
  api:
    build: .
    image: rails_postgres_redis:latest
    command: bash -c "(rake db:exists && rake db:migrate || rake db:setup) && bundle exec rails s -b 0.0.0.0"
    environment:
      REDIS_URL: redis://redis:6379/0
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
      POSTGRES_HOST: db
    ports:
      - "3000:3000"
    depends_on:
      - db
  
  db:
    image: postgres:12-alpine
    ports:
      - "5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres

When you docker-compose up, compose will spin up db first and then api second. I want to translate this sort of behavior to Cue. My very rough sketch for this where I'm only focusing on the name of the service and the depends on looks something like

#ErrorMessage: {
  message: "Something went wrong"
}

#Service: {
  name: string,
  success: bool
}

#Compose: {
  services: [...#Service | #ErrorMessage]
}

api: #Service & {
  name: "api"
}

db: #Service & {
  name: "db"
}

myCompose:  #Compose & {
  services: [
    db,
    if (db.success) {
      api
    }
    if (!db.success) {
      #ErrorService
    }
  ]
}

Now the part I'm running into a mental block with how to get the full output from myCompose. cue eval -c fails because there is incomplete data, db.success is has not been defined. In this contrived example I could input that value, but in a more complex branching structure where there are many levels of depends on, I'm struggling to understand how I could paint a full picture of all the possibilities.

Imagine wanting to generate a graph showing the different paths / outcomes to a user based on the definition of their Cue file or translating the services into database objects with proper relations on what to run next if something is successful.

[verdverm]

I'm not sure you will be able to replace depends_on in pure CUE.

You should be able use the tooling layer API to do this, and inject a depends_on like field. https://pkg.go.dev/cuelang.org/[email protected]/tools/flow

You might also be able to create a "compose" like tool within CUEs tooling layer. cue help cmd CUE's tooling layer has some automattic dependency detection between tasks. This tooling layer has imperative constructs, whereas "pure" CUE does not.

cc: @shykes @aluzzardi @samalba who have a tool called dagger that might be of interest

As far as printing possible concrete configurations from an incomplete value, that sounds like a complex problem with possibly infinite solutions. You might see more desirable output if you remove the concreteness requirement during cue eval, but I suspect it will not reach your desired sophistication.

[jer-k]

Thanks for the input (again!). I actually learned about Cue through @shykes DockerCon presentation on Dagger so that is where my exploration comes from. If you recall from my other question (#1190) I'm kind of trying to figure out how to replicate a Github Actions type workflow, not actually a Compose replacement. I just happened to use Compose's depends_on here as I thought it provided a solid concrete example.

The flow tooling is interesting, I'm going to look into that more. It might be that using it and walking through the tree is the result I'm looking for.

[verdverm]

You could copy the code over and/or strip out the dependency detection algo if you do not need the execution engine. There is a nice DAG processor in there. You may still need to allow for explicit dependency denotation like TF has, when it cannot be inferred automatically. There are some known issues with cue/flow's detection, notably the if guard. There is at least one open issue on this.

Have you looked into Attributes? They allow for markup that is not processed by CUE's evaluator, and are largely intentioned for tools building on CUE

[samalba]

Thanks for the input (again!). I actually learned about Cue through @shykes DockerCon presentation on Dagger so that is where my exploration comes from. If you recall from my other question (#1190) I'm kind of trying to figure out how to replicate a Github Actions type workflow, not actually a Compose replacement. I just happened to use Compose's depends_on here as I thought it provided a solid concrete example.

Thanks @verdverm for the ping. @jer-k if you need access to dagger or need a quick demo to help you out, just let me know.

[jer-k]

@samalba I would love to check out dagger and learn more about what you all are up to! There is an email on my github profile or what would be the best way to contact?

[samalba]

@jer-k you can just fill out https://dagger.io/ - mention this conversation at the end and I'll give you access today.