diff --git a/README.md b/README.md index de25e2c77..da4365101 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,9 @@ the [Kotlin Source Code Repository](https://github.com/JetBrains/kotlin). ## Current KEEPs * Current in-progress KEEPs are listed in [issues](https://github.com/Kotlin/KEEP/issues). + Each KEEP is assigned an ID. KEEP ID is the same as issue ID * New KEEPs and additions to current KEEPs are submitted as [pull requests](https://github.com/Kotlin/KEEP/pulls). + Every KEEP file name must be prefixed with the KEEP ID * When KEEPs are implemented, the corresponding design documents are merged into this repository and stored in a [proposals](proposals) directory. ### Design notes diff --git a/proposals/type-aliases.md b/proposals/KEEP-0004-type-aliases.md similarity index 100% rename from proposals/type-aliases.md rename to proposals/KEEP-0004-type-aliases.md diff --git a/proposals/bound-callable-references.md b/proposals/KEEP-0005-bound-callable-references.md similarity index 100% rename from proposals/bound-callable-references.md rename to proposals/KEEP-0005-bound-callable-references.md diff --git a/proposals/local-delegated-properties.md b/proposals/KEEP-0025-local-delegated-properties.md similarity index 100% rename from proposals/local-delegated-properties.md rename to proposals/KEEP-0025-local-delegated-properties.md diff --git a/proposals/sealed-class-inheritance.md b/proposals/KEEP-0029-sealed-class-inheritance.md similarity index 100% rename from proposals/sealed-class-inheritance.md rename to proposals/KEEP-0029-sealed-class-inheritance.md diff --git a/proposals/jdk-dependent-built-ins.md b/proposals/KEEP-0030-jdk-dependent-built-ins.md similarity index 100% rename from proposals/jdk-dependent-built-ins.md rename to proposals/KEEP-0030-jdk-dependent-built-ins.md diff --git a/proposals/data-class-inheritance.md b/proposals/KEEP-0031-data-class-inheritance.md similarity index 100% rename from proposals/data-class-inheritance.md rename to proposals/KEEP-0031-data-class-inheritance.md diff --git a/proposals/destructuring-in-parameters.md b/proposals/KEEP-0032-destructuring-in-parameters.md similarity index 100% rename from proposals/destructuring-in-parameters.md rename to proposals/KEEP-0032-destructuring-in-parameters.md diff --git a/proposals/inline-properties.md b/proposals/KEEP-0034-inline-properties.md similarity index 100% rename from proposals/inline-properties.md rename to proposals/KEEP-0034-inline-properties.md diff --git a/proposals/explicit-api-mode.md b/proposals/KEEP-0045-explicit-api-mode.md similarity index 100% rename from proposals/explicit-api-mode.md rename to proposals/KEEP-0045-explicit-api-mode.md diff --git a/proposals/underscore-for-unused-parameters.md b/proposals/KEEP-0055-underscore-for-unused-parameters.md similarity index 100% rename from proposals/underscore-for-unused-parameters.md rename to proposals/KEEP-0055-underscore-for-unused-parameters.md diff --git a/proposals/scope-control-for-implicit-receivers.md b/proposals/KEEP-0057-scope-control-for-implicit-receivers.md similarity index 100% rename from proposals/scope-control-for-implicit-receivers.md rename to proposals/KEEP-0057-scope-control-for-implicit-receivers.md diff --git a/proposals/underscores-in-numeric-literals.md b/proposals/KEEP-0059-underscores-in-numeric-literals.md similarity index 100% rename from proposals/underscores-in-numeric-literals.md rename to proposals/KEEP-0059-underscores-in-numeric-literals.md diff --git a/proposals/generic-values-and-valueof-for-enums.md b/proposals/KEEP-0061-generic-values-and-valueof-for-enums.md similarity index 100% rename from proposals/generic-values-and-valueof-for-enums.md rename to proposals/KEEP-0061-generic-values-and-valueof-for-enums.md diff --git a/proposals/KEEP-0075-scripting-support.md b/proposals/KEEP-0075-scripting-support.md new file mode 100644 index 000000000..7903582e6 --- /dev/null +++ b/proposals/KEEP-0075-scripting-support.md @@ -0,0 +1,895 @@ + +# Kotlin Scripting support + +*Replaces [Script Definition Template](https://github.com/Kotlin/KEEP/blob/master/proposals/script-definition-template.md) +proposal.* + +## Feedback + +Discussion of this proposal is held in [this issue](https://github.com/Kotlin/KEEP/issues/75) + +## Motivation + +- Define Kotlin scripting and its applications +- Describe intended use cases for the Kotlin scripting +- Define scripting support that is: + - applicable to all Kotlin platforms + - provides sufficient control of interpretation and execution of scripts + - simple enough to configure and customize + - provides usable default components and configurations for the typical use cases +- Provide basic examples of the scripting usage and implementation +- Address the issues found during the public usage of the current scripting support + +## Status of this document + +The document is still a draft, and few important parts are still missing, in particular: + +- REPL: + - API for REPL host configuration and embedding + - API for REPL plugins (maybe should be covered elsewhere): + - custom repl commands + - highlighting + - completion + - etc. +- IDE plugins API for custom scripting support (besides discovery) + +## Table of contents + +- [Applications](#applications) +- [Basic definitions](#basic-definitions) +- [Use cases](#use-cases) + - [Embedded scripting](#embedded-scripting) + - [Standalone scripting](#standalone-scripting) + - [Project infrastructure scripting](#project-infrastructure-scripting) + - [Script-based DSL](#script-based-dsl) +- [Proposal](#proposal) + - [Architecture](#architecture) + - [Script definition](#script-definition) + - [Script Compilation](#script-compilation) + - [Script Evaluation](#script-evaluation) + - [Standard hosts and discovery](#standard-hosts-and-discovery) + - [How to implement scripting support](#how-to-implement-scripting-support) + - [Implementation status](#implementation-status) + - [Examples](#examples) + - [kotlin-main-kts](#kotlin-main-kts) + +## Applications + +- Build scripts (Gradle/Kobalt) +- Test scripts (Spek) +- Command-line utilities +- Routing scripts (ktor) +- Type-safe configuration files (TeamCity) +- In-process scripting and REPL for IDE +- Consoles like IPython/Jupyter Notebook +- Game scripting engines +- ... + +## Basic definitions + +- **Script** - a text file written in Kotlin language but allowing top-level statements and expressions and having access + to some implicit (not directly mentioned in the script text) properties, functions and objects, as if the whole script + body is a body of an implicit function placed in some environment (see below) +- **Scripting Host** - an application or a component which handles script execution +- **Scripting Host Environment** - a set of parameters that defines an environment for all scripting host services, + contains when relevant: project paths, jdk path, etc. It is passed on constructing the services +- **REPL snippet** - a group of script text lines, executed in a single REPL eval call +- **Script compilation configuration** - a set of parameters configuring script compilation, such as dependencies, + external variables declarations, implicit import statements, etc. +- **Script evaluation configuration** - a set of parameters configuring script evaluation, such as external variables + instances, actual script parameters, etc. +- **Script definition** - a set of parameters and configurations defining a script type +- **Compiled script** - a binary compiled code of the script, stored in memory or on disk, which could be loaded + and instantiated by appropriate platform +- **Dependency** - an external library or another project whose declarations are available for the script being compiled + and evaluated +- **Imported script** - another script whose declarations are available for the script being compiled and evaluated +- **Configuration refinement** - the process of updating script compilation and evaluation configurations during + compilation/evaluation with parameter specific to the particular script exemplar, e.g. depending on the script + contents + +## Use cases + +### Embedded scripting + +The use case when a scripting host is embedded into user's application, e.g. specialized console like +IPython/Jupyter notebook, Spark shell, embedded game scripting, IDE and other application-level scripting. + +#### Environment and customization + +In this case the script is most likely need to run in a specific execution environment, defined by the scripting host. +The default script compilation and evaluation configurations are defined by the scripting host as well. The host may +provide script authors with a possibility to customize some configuration parameters, e.g. to add dependencies or +specify additional compilation options, e.g. using annotations in the script text: + +```kt +@file:DependsOn("maven:artifact:1.0", "imported.package.*") +@file:Import("path/to/externalScript.kts") +@file:CompilerOptions("-someCompilerOpt") +``` + +To implement this the compilation/evaluation pipeline should provide a callback mechanism to call a host-provided code +that could analyze script text directly or parsed annotations in order to update compilation and evaluation +configurations with parameters depending on the script contents or other dynamic factors. + +If the host need to support several script *types*, i.e. sets of compilation configurations and customization means, +there should be a way for the host to distinguish the scripts and select appropriate set of +compilation/configuration/evaluation services and properties. The host authors can implement the selection based on any +script property, but due to the difficulties in supporting file type distinction based on anything but filename +extension across platforms and IDEs, the default implementations should support only extension-based or file name based +selection. + +*Note: It would be nice to provide an infrastructure (complete hosts or libraries) that support some typical mean for +each platform to resolve external libraries from online repositories (e.g. - maven for JVM) out of the box.* + +#### Typical usages + +In a simple case, the developer wants to implement a scripting host to control script execution and provide the required +environment. One may want to write something as simple as: +```kt +KotlinScriptingHost().eval(File("path/to/script.kts")) +``` +or +```kt +KotlinScriptingHost().eval("println(\"Hello from script!\")") +``` +and the script should be executed in the current environment with some reasonable default compilation and evaluation +settings. If things need to be configured explicitly, the code would look like: +```kt +val scriptingHost = KotlinScriptingHost(configurationParams...) +scriptingHost.eval(File("path/to/script.kts"), compilationConfiguration, evaluationConfiguration) +``` +This would also allow the developer to control the lifetime of the scripting host. + +In cases then there is more control needed for the compilation and evaluation process, the direct use of the script +compiler and evaluator could be desirable: +```kt +val scriptCompiler = KotlinScriptCompiler(configurationParams...) +val compiledScript = scriptCompiler.invoke(File("path/to/script.kts"), compilationConfiguration) +val scriptEvaluator = KotlinScriptEvaluator(configurationParams...) +val evaluationResult = scriptEvaluator.invoke(compiledScript, evaluationConfiguration) +``` +In addition in this case the compiled script could be evaluated more than once. + +#### More control + +To be able to run scripts in a user controlled environment, the following information bits could be configured or +provided to the host: +- *Script Compiler* - the service that will compile scripts into a form accepted by the Evaluator +- *Script Compilation Configuration* - set of properties defining the script compilation settings, including: + - *Script Base Class* - an interface (or prototype) of the script class, expected by the executor, so the compiler + should compile script into the appropriate class + - *Dependencies* - external libraries that could be used in the script + - *Default imports* - import statements implicitly added to any compiled script + - *Provided properties* - properties with types that is assumed visible in the script scope + - etc. +- *Script Evaluator* - the service that will actually evaluate the compiled scripts in the required execution + environment +- *Evaluation configuration* - set of properties defining an environment for the evaluation, including: + - *Provided properties* - actual values of the provided properties from the compilation configuration + - etc. + +Some of these parameters could be wrapped into a *Script Definition* for easier identification of the script types +by the hosts that may support handling of the several script types simultaneously. + +#### Caching + +Since calling Kotlin compiler could be quite a heavy operation in comparison with typical script execution, the caching +of the compiled script should be supported by the compilation platform. + +#### Execution lifecycle + +The script is executed according to the following scheme: +- compilation - the *Script Compiler* takes the script, its compilation configuration and + provides a compiled class. Inside this process. If the compilation configuration defines refinement callbacks, they + are called before or after parsing to update configuration with parameter depending on the script contents or other + dynamic data. +- evaluation - the *Script Evaluator* takes the compiled script instantiates it if needed, and calls the appropriate + method, passing arguments from the environment to it; this step could be repeated many times. If evaluation + configuration defines refinement callbacks, they will be called before evaluation similarly to the compilation ones. + +#### Processing in an IDE + +The IDE support for the scripts should be based on the same *Script definition*. Basically after recognizing the script +*type* (see *Environment and customization* section above), the IDE extracts the *Compilation Configuration* and use its +parameters to implement highlighting, navigation and other features. The default implementation of Kotlin IDEA plugin +should support the appropriate functionality, based on the standard set of configuration parameters. + +### Standalone scripting + +Standalone scripting applications include command-line utilities and a standalone Kotlin REPL. + +Standalone scripting is a variant of the embedded scripting with hosts provided with the Kotlin distribution or +3rd-party hosts. + +#### Hosts + +The standalone script could be executed e.g. using command line Kotlin compiler: + +```sh +kotlinc -cp -script myscript.kts +``` + +Or with the dedicated runner included into the distribution: + +```sh +kotlin -cp myscript.kts +``` + +To be able to use the Kotlin scripts in a Unix shell environment, the *shebang* (`#!`) syntax should be supported +at the beginning of the script: +```sh +#! /path/to/kotlin/script/runner -some -params +``` + +*Note: due to lack of clear specification, passing parameters in the shebang line could be +[problematic](https://stackoverflow.com/questions/4303128/how-to-use-multiple-arguments-with-a-shebang-i-e/4304187#4304187), +therefore alternative schemes of configuring scripts should be available.* + +#### Script customizations + +It should be possible to process custom scripts with the standard hosts, e.g. by supplying a custom script definition +in the command line, e.g.: +```sh +kotlin -script-templates="org.acme.MyScriptDef" -cp myScriptDefLib.jar myscript.myscr.kts +``` +In this case the host loads specified definition class, and extract required definition from it and its annotations. + +Another possible mechanism is automatic discovery, simplifying usage: +```sh +kotlin -cp myScriptDefLib.jar myscript.myscr.kts +``` +In this case the host analyses the classpath, discovers script definitions located there and then processes then as +before. Note that in this case it is should be recommended to use dedicated script extension (`myscr.kts`) in every +definition to minimize chances of clashes if several script definitions will appear in the classpath. And on top of +that, some selection mechanism based on the file names and/or paths is needed, e.g. to cover cases like gradle's +`build.gradle.kts` and `setting.gradle.kts` that should be treated like different script types. + +#### Script parameters + +For the command line usage the support for script parameters is needed. The simplest form is to assume that the script +has access to the `args: Array` property/parameter. More advanced is to have a customization that supports a +declaration of the typed parameters in the script annotations e.g.: + +```kt +@file:param("name", "String?") // note: stringified types are used for the cases not supported by class literals +@file:param("num", Int::class) +@file:param("list", "List") + +// this script could be called with args "-name=abc -num=42 -list=a,b,c" +// and then in the body we can access parsed typed arguments + +println("${name ?: ""} ${num/6}: ${list.map { it.toUpperCase() } }") +``` + +#### IDE support + +Since in this use case scripts are not part of any project, support for such script should be configured in the IDE +explicitly, either via a plugin or via the IDE settings. + +#### Standalone REPL + +Standalone REPL is invoked by a dedicated host the same way as for standalone script but accepts user's input as repl +snippets. It means that the declarations made in the previous snippets are accessible in the subsequent ones. +In this mode, all new scripting features should be accessible as well, including customization. + +### Project infrastructure scripting + +Applications: project-level REPL, build scripts, source generation scripts, etc. + +Project infrastructure scripts are executed by some dedicated scripting host usually embedded into the project build +system. So it is a variant of the embedded scripting with the host and the IDE support integrated into build system +and/or IDE itself. + +#### IDE support + +From an IDE point of view, they are project-context dependent, but may not be part of the project sources. (In the same +sense as e.g. gradle build scripts source is not considered as a part of the project sources.) In this case the support +in the IDE is possible only if the definitions are supplied to the IDE explicitly, similarly to standalone scripts, +either via a plugin or via the IDE settings. + +#### Discovery + +The IDE needs to be able to extract scripts environment configurations from the project settings, if the scrips are +considered a part of the project sources, so the project's classpath could be used for discovery of the supported +scripts. + +#### Project-level REPL + +A REPL that has access to the project's compiled classes. + +### Script-based DSL + +Applications: test definition scripts (Spek), routing scripts (ktor), type safe config files, etc. + +In these cases the scripts are considered parts of the kotlin project and are compiled to appropriate binary form by the +compiler, and then linked with the rest of the compilation results. They differ from the other project's sources by the +possibility to employ script semantic and configurability and therefore avoid some boilerplate and make the sources +look more DSL-like. + +In this scenario, no dedicated scripting host is used, but the standard compiler is used during the regular compilation +according to configured script recognition logic (e.g. the script type discovery mechanism described above), and the +target application should implement its own logic for instantiating and calling the generated script classes. +Script compiler may also annotate generated classes and methods with user-specified annotations to integrate it with +existing execution logic. E.g. junit test scripts could be annotated accordingly. + +From an IDE point of view, these scripts are the part of the project but should be configured according to the +recognized script definition. + +## Proposal + +### Architecture + +#### Components + +The scripting support consists of the following components: +- `ScriptCompiler` - interface for script compilation + - compilation: `(scriptSource, compilationConfiguration) -> compiledScript` + - predefined script compilers based on the kotlin platforms: /JVM, /JS, /Native + - custom/customized implementation possible + - compiled scripts caсhing belongs here + - should not keep the state of the script compilation, the required state for the subsequent compilations, e.g. in the + REPL mode, is passed along with the compiled script +- `ScriptEvaluator` - the component that receives compiled script instantiates it, and then evaluates it in a required + environment, supplying any arguments that the script requires: + - evaluation: `(compiledScript, evaluationConfiguration) -> Any?` + - the `compiledScript` contains the final compilation configuration used + - the `evaluationConfiguration` the parameters describing the actual script evaluation configuration of the script + - predefined platform-specific evaluators available, but could be provided by the scripting host + - pseudo-evaluators that save compiled script into e.g. an executable jar should be provided as well +- **IDE support** - Kotlin IDEA plugin should have support for scripting with script definition selection based on the + file name extension, and also includes discovery. The exposed generic ide support that would allow to build rich + script editing apps and REPLs is outside of the scope of this proposal and will be covered elsewhere. + +#### Data structures + +- `ScriptSource` determines the way to access script for other components; it consists of: + - the script reference pointer: url + - an accessor to the script text + Both components are optional but at least one is required for a regular script. + The class implements source position and fragment referencing classes used e.g. in error reporting. +- `KotlinType` a wrapper around Kotlin types, used to decouple script definition and compilation/evaluation + environments. It could be constructed either from reflected or from stringified type representation. +- `ScriptCompilationConfiguration` - a heterogeneous container of parameters defining the script compilation +- `ScriptEvaluationConfiguration` - a heterogeneous container of parameters defining the script evaluation +- `ScriptingHostConfiguration` - a heterogeneous container of host-specific parameters + +### Script definition + +Script Definition is a way to specify custom script. It is basically consists of a script base class annotated with +`KotlinScript` annotation. The arguments of the annotation define the script configuration parameters. For example: + +```kt +@KotlinScript( + displayName = "My Script", + fileExtension = "myscr.kts", + compilationConfiguration = MyScriptCompilationConfiguration::class, + evaluationConfiguration = MyScriptEvaluationConfiguration::class +) +abstract class MyScript(project: Project, val name: String) { + fun helper1() { ... } + + [@ScriptBody] + [suspend] abstract fun (params...): MyReturnType +} + +object MyScriptCompilationConfiguration : ScriptCompilationConfiguration({ + defaultImports("java.io.*") + providedProperties(prop1 to Int::class, prop2 to String::class) +}) + +object MyScriptEvaluationConfiguration : ScriptEvaluationConfiguration({ + providedProperties(prop1 to 42, prop2 to "foo") +}) +``` + +*Where:* +- any valid method name could be used in place of `` +- `@ScriptBody` annotation marks the method the script body will be compiled into. In the absence of the explicit + annotation, if the configuration requires compilation into a method, the SAM notation will be used, otherwise + the script will be generated into the target class constructor. +- `interface` or `open class` could be used in place of the `abstract class` +- *(see also possible compilation configuration properties below)* + +The annotations have reasonable defaults, so in the minimal case it is enough to mark the class only with the +`@KoltinScript` without parameters. But it is recommended to give a dedicated file name extension for every script +type to minimize chances for clashes in case of multiple definitions in one context. +The file name extension could be declared in the compilation configuration as well, but it is highly recommended +to define it in the annotation instead, since it will speed up the discovery process significantly. + +### Script Compilation + +#### Script Compiler + +Script compiler implements the following interface: +```kt +interface ScriptCompiler { + + /** + * Compiles the [script] according to the [scriptCompilationConfiguration] + * @param script the interface to the script source code + * @param scriptCompilationConfiguration the script compilation configuration properties + * @return result wrapper, if successful - with compiled script + */ + suspend operator fun invoke( + script: SourceCode, + scriptCompilationConfiguration: ScriptCompilationConfiguration + ): ResultWithDiagnostics> +} +``` +where: +```kt +interface CompiledScript { + + /** + * The compilation configuration used for script compilation + */ + val compilationConfiguration: ScriptCompilationConfiguration + + /** + * The function that loads compiled script class + * @param scriptEvaluationConfiguration the script evaluation configuration properties + * @return result wrapper, if successful - with loaded KClass + */ + suspend fun getClass(scriptEvaluationConfiguration: ScriptEvaluationConfiguration?): ResultWithDiagnostics> + + /** + * The name and the type of the script's result field, if any + */ + val resultField: Pair? +} + +``` +The compilers for the supported platforms are supplied by default scripting infrastructure. + +#### Script Dynamic Compilation Configuration + +The script compilation could be configured dynamically by specifying configuration refining callbacks in the compilation +configuration. The callback should be written according to the following signature: +```kt +typealias RefineScriptCompilationConfigurationHandler = + (ScriptConfigurationRefinementContext) -> ResultWithDiagnostics +``` +where: +```kt +class ScriptConfigurationRefinementContext( + val script: SourceCode, + val compilationConfiguration: ScriptCompilationConfiguration, + val collectedData: ScriptCollectedData? = null +) +``` +and `ScriptCollectedData` is a heterogeneous container of properties with the appropriate data collected during parsing. + +See the following section for the parameters that declare such callbacks. + +#### Compilation Configuration Properties + +The following properties are recognized by the compiler: +- `baseClass` - target script superclass as well as a source for the script target method signature, constructor + parameters and annotations, default - `Any`. +- `sourceFragments` - script fragments compile - allows to compile script partially +- `scriptBodyTarget` - defines whether script body will be compiled into resulting class constructor or to a + method body. In the latter case, there should be either single abstract method defined in the script base class, or + single appropriate method should be annotated with the `ScriptBody` annotation +- `implicitReceivers` - a list of script types that is assumed to be implicit receivers for the script body, as + if the script is wrapped into `with` expressions, in the order from outer to inner scope, i.e.: + ```kt + with(receiver0) { + ... + with(receiverN) { +