import/include another cue-file (using a relative path)

[cueckoo]

Originally opened by @bergkvist in cuelang/cue#836

Is your feature request related to a problem? Please describe.
I want to be able to import/include other cue-files directly using a relative path. Example:

// schema.cue
a: number & >10
b: *10 | number

// first.cue
include "./schema.cue"
a: 12

// second.cue
include "./schema.cue"
a: 13
b: 11

Then I could run:

$ cue eval first.cue
a: 12
b: 10

$ cue eval second.cue
a: 13
b: 11

to produce two different outputs that share something in common.

---

In particular I want to use this pattern with Kubernetes configs to create something like:

• staging.cue
• production.cue
• deployment.cue

Where staging.cue and production.cue both include deployment.cue.

I don't want production.cue and staging.cue to be loaded at the same time.

The include-statement could be implemented like "#include" in the C-preprocessor, simply inserting the contents of the included file directly into where the include-keyword was used.

I think one of the main issues with "#include" in the C-preprocessor was that it is not idempotent (by default anyways), and that order matters. It seems to me like neither of these things are issues in Cue, since definitions are idempotent and order-independent.

[cueckoo]

Original reply by @verdverm in cuelang/cue#836 (comment)

related: cuelang/cue#403

This may be helpful as an alternative way to do this: https://cuelang.org/docs/concepts/packages/#instances

[cueckoo]

Original reply by @bergkvist in cuelang/cue#836 (comment)

Thanks @verdverm

So I have to create a cue-module to import other local cue-files?

Ideally I'd want my folder structure to reflect what I'm trying to accomplish, rather than which tools I'm using. It seems like cue modules force a very opinionated folder structure.

[cueckoo]

Original reply by @verdverm in cuelang/cue#836 (comment)

The above is one possible way to accomplish what you'd like among many. You can organize your directories any way you like with imports. To import across directories, a boundary to the import paths is needed (a.k.a. setting up a module, which is basically just the cue.mod/module.cue file). See the link to the Go discussion on #403 why relative paths are not supported.

I wrote an example here that shows some other ways: https://cuetorials.com/first-steps/modules-and-packages/

If you don't want to use packages, you should be able to list all paths to be merged at the command line.

[cueckoo]

Original reply by @mpvl in cuelang/cue#836 (comment)

Ideally I'd want my folder structure to reflect what I'm trying to accomplish, rather than which tools I'm using.
What do you mean with which tools you're using?

It seems like cue modules force a very opinionated folder structure.

We are thinking about expanding the importing possibilities a bit. However, in order for CUE to remain meaningfully hermetic, we would still require a cue module. The main purpose of the module is to mark some root. There is not much more to it.

We are considering allowing relative paths for individual files (non-package CUE, JSON or YAML), but in this case it seems that the standard package mechanism would work. Also, keep in mind that for the relative imports, we would still need a CUE module to deal with versioning. Not just for CUE. YAML, for instance, has a history of introducing backwards-incompatible changes and even allows deviations within a single version. The module allows us to add any meta-interpretation needed to properly interpret YAML. Even JSON can have compatibilities issues.

We do intend to keep modules light-weight, though, so that this does not need to be burdensome.

[cueckoo]

Original reply by @bergkvist in cuelang/cue#836 (comment)

The way it works is not really intuitive to me. I can't figure out how to import another cue file in the same folder - even after wrapping the two files in a module.

What I've tried:

cue mod init example.com
echo "b: 2" > cue.mod/usr/b.cue
echo 'import "example.com/usr/b"\na: 1' > cue.mod/usr/a.cue
cue eval cue.mod/usr/a.cue
# import failed: cannot find package "example.com/usr/b":
#     ./cue.mod/usr/a.cue:2:8

Expected output:

a: 1
b: 2

Trying to add .cue extension to the import statement to see if it makes a difference:

echo 'import "example.com/usr/b.cue"\na: 1' > cue.mod/usr/a.cue
cue eval cue.mod/usr/a.cue
# import failed: cannot determine package name for "example.com/usr/b.cue" (set explicitly with ':'):
#     ./cue.mod/usr/a.cue:2:8

Okay, what is this package-thing? and why do I need it?
I'll try to add "package root" to the top of all of the files, and ":root" to the import statement to see if it helps.

echo 'package root\nb: 2' > cue.mod/usr/b.cue
echo 'package root\nimport "example.com/usr/b.cue:root"\na: 1' > cue.mod/usr/a.cue
cue eval cue.mod/usr/a.cue
# import failed: cannot find package "example.com/usr/b.cue:root":
#     ./cue.mod/usr/a.cue:2:8

import formats I've tried:

import "example.com/usr/b" # import failed: cannot find package "example.com/usr/b"
import "example.com/usr/b:root" # import failed: cannot determine package name for "example.com/usr/b.cue" (set explicitly with ':')
import "example.com/usr/b.cue:root" # import failed: cannot find package "example.com/usr/b.cue:root"
import "example.com/usr/b:root" # import failed: cannot find package "example.com/usr/b:root"
import "example.com/b" # import failed: cannot find package "example.com/b"
import "example.com/b.cue" # import failed: cannot determine package name for "example.com/b.cue" (set explicitly with ':')
import "example.com/b.cue:root" # import failed: cannot find package "example.com/b.cue:root"
import "example.com/b:root" # import failed: cannot find package "example.com/b:root"

What am I doing wrong here?

Some other questions:

• Does the folder that contains module.cue need to be named "cue.mod"?
• Do I actually need to make up a valid hostname to make a simple config file importable?
• What is pkg/, usr/, gen/? Are these folders special when it comes to the import statement?
• What is the purpose of the package-statement? Do I need it in order for a cue file to be "importable"?

I guess my difficulty getting this to work comes from me not being familiar enough with the module system in Golang, which it seems like Cue depends on.

[cueckoo]

Original reply by @myitcv in cuelang/cue#836 (comment)

@bergkvist have you see the introduction to packages and modules at https://cuelang.org/docs/concepts/packages/?

[cueckoo]

Original reply by @bergkvist in cuelang/cue#836 (comment)

Okay - I managed to at least get something working! A major misconception from my side here:

I should not put my config files inside the cue.mod-folder, but rather next to it!

Some more questions in my head right now:

• I don't seem to be able to import individual *.cue-files, only folders. Is this not supported?
• Do I always need to put a package-statement in the top of each file? What does the "package"-statement really give me?

The most minimal setup I've been able to create is:

(click to expand)

my-project/cue.mod/module.cue

module: example.com

my-project/a.cue

package x
import b "example.com/b:y"
a: 1
b

my-project/b/b.cue

package y
b: 2

And output:

$ cue eval my-project/a.cue
a: 1
b: 2

[cueckoo]

Original reply by @bergkvist in cuelang/cue#836 (comment)

Making some mutations to my-project/a.cue that I expected not to make a difference:

Note that the command run is always cue eval a.cue

// removed "package x"
import b "example.com/b:y"
a: 1
b

^ Seems to work the same way still, which is nice

package y  // changed from x to y
import b "example.com/b:y"
a: 1
b

^ Just hangs forever without doing anything

import b "example.com/b/b.cue:y"  // specified file instead of folder
a: 1
b

^ import failed: cannot find package "example.com/b/b.cue:y"

import "example.com/b:y"  // removing b-variable
a: 1

^ imported and not used: "example.com/b:y"

import b "example.com/b"  // removing package specification
a: 1
b

^ import failed: build constraints exclude all CUE files in example.com/b (ignored: b/b.cue, a.cue)

[cueckoo]

Original reply by @myitcv in cuelang/cue#836 (comment)

I should not put my config files inside the cue.mod-folder, but rather next to it!

Correct. The existence of a cue.mod directory marks the root of a module. i.e. the the directory that contains the cue.mod directory is the root of the module.

I don't seem to be able to import individual *.cue-files, only folders. Is this not supported?

CUE does not support importing of individual CUE files. It might support embedding of individual CUE files, but that's not currently supported and quite a different concept.

CUE heavily relies on its order independence for package organisation. Definitions and constraints can be split across files within a package, and even organised across directories. So in that sense a package abstracts away whether a definition/constraint exists in one file, or is split across multiple (in multiple directories). Reasons like this are why CUE has decided to follow the Go model of abstracting this to the concept of a package.

Do I always need to put a package-statement in the top of each file? What does the "package"-statement really give me?

To be able to import the CUE you are defining, yes. Where you have a CUE file(s) that is not imported elsewhere, you don't strictly need it, but generally it's good practice to think in terms of packages. Because that way the cue command can operate more seamlessly without you always needing to specify an explicit file list on the command line.

Adapting your minimal example slightly:

-- cue.mod/module.cue --
module: "example.com/a"
-- a.cue --
package blah

import "example.com/a/b"

x: 5
y: b.y
-- b/b.cue --
package b

y: 4

(note the format of expressing multiple files like this is txtar, used heavily in CUE, especially CUE's tests)

Within the root of the module you can simply run cue eval - that evaluates the package in the current directory, the package at the import path example.com/a. That package imports example.com/a/b as you can see. The package identified by the import path example.com/a/b is named b (see b/b.cue). Hence you don't need the explicit package name in the import path as you have in your example - where the import path and the package name agree that selector can be elided.

I'm going to migrate this issue to a discussion so that others can benefit from these great questions.

[cueckoo]

Original reply by @bergkvist in cuelang/cue#836 (comment)

I'd love for embedding of individual CUE files to be supported in the future.

Just to clarify what I think you mean when you say embed: I imagine embed "./x.cue" or just embed "./x" to be equivalent to pasting the contents of the file "x.cue" (which is
located in the same folder) into where the embed-keyword was used. Similar to how #include "some-file.h" works in C/C++.

[cueckoo]

Original reply by @myitcv in cuelang/cue#836 (comment)

@bergkvist can you explain your use case a bit more, perhaps with a concrete example? Just trying to understand how the existing code organisation structure/approach is insufficient.

[cueckoo]

Original reply by @eonpatapon in cuelang/cue#836 (comment)

In particular I want to use this pattern with Kubernetes configs to create something like:

* `staging.cue`

* `production.cue`

* `deployment.cue`

Where staging.cue and production.cue both include deployment.cue.

I don't want production.cue and staging.cue to be loaded at the same time.

For this you could use build tags see (cue help injection, "Injecting files").

   // File prod.cue
   @if(prod)

   package xxx
   ...
   
   //File stg.cue
   @if(stg)
   
   package xxx
   
   // File deployment.cue
   package xxx
   
   ...

When evaluating you choose which file (prod.cue or stg.cue) to include:

cue -t prod eval