Golang user-defined functions

[joshuamorris3]

Besides the builtin functions available from the cuelang package, is it possible to call Golang functions from user-defined or other standard library packages. Specifically I would like to use mail.ParseAddress from https://pkg.go.dev/net/mail.

[myitcv]

This sort of extensibility will be covered by the WebAssembly (Wasm) work that we are doing: #2463. Such Wasm modules will then be distributed via modules (#2330). In this way, people will not be limited by the standard library of builtins that we (currently) offer.

That said, there are often strong arguments for making something part of the standard library. I would suggest that mail.ParseAddress comes pretty close to passing an as-yet-undefined threshold for inclusion. Please can you raise a feature request to that end if you would like this added? It feels ok to me, but a feature request will help to confirm.

(Incidentally, we also somewhat envisage a Go-like cuelang.org/x/... namespace of modules for "things that aren't quite standard library but are still useful". That's intentionally a very loose category and definition, but something of a recognition that the import path for a module (with Wasm modules) matters from a provenance perspective).

cc @4ad for more information on the Wasm side of things.

[thatInfrastructureGuy]

@myitcv since WASM development is paused, is there a path forward for user-defined Golang functions?

[myitcv]

@thatInfrastructureGuy - we have, in effect, let this point drop through the cracks. Thank you for pointing it out. I have it on my TODO list to ensure that we create a new issue that pulls together the various threads of discussion, issues etc, with a clear statement of where we are (including with respect to WASM), and given the current state of WASM work what our plan b could/should be.

[myitcv]

Just bumping with a response here to confirm we are actively working on providing a substantive update here.

[verdverm]

The thought crossed my mind that another plan b would be the "workflows in core evaluator" idea, where the evaluator is smart enough to detect and order things appropriately ( I recall this concept being discussed on a community call some time ago...)

I think it is spiritually similar to wasm, just a different "backend" implementation. Users could write "custom functions" that leverage exec.Exec to implement using pretty much anything

One long desired feature is the "function syntax", where we can write val: foo(...) (originally for wrapping #Defs) but could have plain CUE, wasm, and/or workflows; such that the caller doesn't know or care.

[joshuamorris3]

Thank you. I've opened a feature request here #2499

[myitcv]

Re-opening this discussion given #2498 (reply in thread) (because I keep trying to find this discussion and failing because it's closed!)

[gotwarlost]

Tangentially related in that this should never be a native builtin - I have really, really wanted to match Kubernetes labels against a label selector (using the labels package in k8s). This would really simplify our cue-crossplane compositions by being able to conditionally set attributes based on object labels. Right now, we have to do some preprocessing elsewhere and hand the results to cue.

[rajikak]

Hello,

Are you working to enable the ability to use, user defined functions ? I see an update in #2498 (reply in thread), but not sure if you are referring to adding mail.ParseAddress to the standard library or adding the support for enabling user defined functions/libraries.

I am working on a DSL based on CUE language. A connection to a SFTP server is defined as:

#Connection: {
	Protocol:  "SFTP" | "FTP" | "FTPS" | "SCP" | "SSH"
	Host:      string 
	Port:      int 
	Username?: string
	Password?: string
}

so I can declare a connection as:

sftpConn: #Connection & {
        Protocol:  "SFTP"
	Host:      "localhost" 
	Port:      22 
	Username: "username"
	Password: "password"
}

This works, but I don't want sensitive details go into the version control. So I would like to read username and password from the environment. So I defined another type as:

#EnvParam: { 	
	Name: string
}

With that I updated #Connection as:

#Connection: {
	Protocol:   "SFTP" | "FTP" | "FTPS" | "SCP" | "SSH"
	Host:      string 
	Port:      int 
	Username?: string | #EnvParam
	Password?: string | #EnvParam
}

and I can declare a connection without sensitive details:

sftpConn: #Connection & {
        Protocol:  "SFTP"
	Host:      "localhost" 
	Port:      22 
	Username: #EnvParam & {
            Name:  "SFTP_USERNAME"
        }
	Password: #EnvParam & {
             Name:  "SFTP_PASSWORD"
        }
}

This works, but I am thinking if I can import a user defined library this can be more readable:

import "userlib"

sftpConn: #Connection & {
        Protocol:  "SFTP"
	Host:      "localhost" 
	Port:      22 
	Username:  userlib.GetEnv("SFTP_USERNAME")
	Password:  userlib.GetEnv("SFTP_PASSWORD")
        }
}

Thanks for your work in CUE language.

[mvdan]

You can accomplish this with tool/os.Getenv and cue cmd, as you can see here: https://github.com/cue-lang/cue/blob/f2775f863f4cdeea888aa9cca4ee7273a6f2d2f1/cmd/cue/cmd/testdata/script/cmd_env.txtar

We don't have examples in the rendered docs, but the docs are at https://pkg.go.dev/cuelang.org/go/pkg/tool/os too.

[rajikak]

Thanks for the quick reply. I am aware of tool package. I am interested in knowing in how can I import my own libraries, so that I can reuse work from other Go libraries/packages (I am using CUE Go API). For example https://pkg.go.dev/github.com/zclconf/go-cty/cty/function/stdlib (https://github.com/hashicorp/terraform/blob/main/internal/lang/functions.go#L64-L180).

[mvdan]

Calling your own Go functions from CUE is currently not possible. We continue to have plans in this space but we are currently focused on the new evaluator and performance (#2850). If cue cmd is not a possibility for you, then the best option right now is to write a Go program combining your own code with the use of our Go APIs at https://pkg.go.dev/cuelang.org/go/cue.