From 2ef62cd0d5c72e98aecca78fd1d5eb4475588e0a Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Thu, 24 Oct 2024 09:24:41 -0300 Subject: [PATCH 01/16] #991: Added Design Doc and Requirements Doc for Script Options parser --- exaudfclient/docs/script_options_design.md | 148 ++++++++++++++++++ .../docs/script_options_requirments.md | 124 +++++++++++++++ 2 files changed, 272 insertions(+) create mode 100644 exaudfclient/docs/script_options_design.md create mode 100644 exaudfclient/docs/script_options_requirments.md diff --git a/exaudfclient/docs/script_options_design.md b/exaudfclient/docs/script_options_design.md new file mode 100644 index 00000000..f45687a0 --- /dev/null +++ b/exaudfclient/docs/script_options_design.md @@ -0,0 +1,148 @@ +# Design Document + +This document details the design aspects of the new Exasol UDF Client Script Options parser, based on the high-level requirements outlined in the System Requirement Specification. + +## Introduction + +The purpose of this document is to provide an in-depth implementation view of the UDF Client Script Options parser, ensuring it meets all specified high-level requirements and integrates seamlessly into the existing Exasol environment. The parser needs to be implemented in C++. + +## Constraints + +- The parser implementation must be in C++. +- The chosen parser implementation is [ctpg](https://github.com/peter-winter/ctpg), which supports the definition of Lexer and Parser Rules in C++ code. + +## Design Requirements + +This section outlines the specific design requirements linked to the high-level requirements, detailing the solutions and methods to achieve them. + +### Parser Implementation +`dsn~parser-implementation~1` + +Implementation of the parser using [ctpg](https://github.com/peter-winter/ctpg), an open-source parser library. This library will be used to define Lexer and Parser Rules in C++ code, ensuring no additional runtime dependencies exist. + +Needs: req +Covers: +- `req~new-parser-implementation~1` + +### Lexer and Parser Rules +`dsn~lexer-parser-rules~1` + +Define the Lexer rules to tokenize `%optionKey`, `optionValue`, and allowed whitespace characters, including `\t`, `\v`, and `\f`. The Parser rules will define the grammar to correctly identify Script Options, manage multiple options with the same key, and handle duplicates. + +Needs: req +Covers: +- `req~general-script-options-parsing~1` + +### Handling Multiple and Duplicate Options +`dsn~handling-multiple-duplicate-options~1` + +Create a mechanism within the parser to collect and aggregate multiple Script Options with the same key. Use a data structure (e.g., a map of lists) to store options by key and ensure duplicates are handled according to specified rules for different options. + +Needs: req +Covers: +- `req~multiple-duplicate-options-management~1` + +### Script Option Removal Mechanism +`dsn~script-option-removal-mechanism~1` + +Implement a method to remove identified Script Options from the original script code. This method will traverse the code, identify Script Options using defined tokens, and replace them with whitespace to ensure the cleaned script executes smoothly. + +Needs: req +Covers: +- `req~script-option-removal~1` + +### Java %scriptclass Option Handling in Design +`dsn~java-scriptclass-option-handling~1` + +Design logic to correctly identify a single %scriptclass option within the script. Additional occurrences of %scriptclass will be removed from the script code. Use a flag to mark the first occurrence and ensure subsequent ones are discarded. + +Needs: req +Covers: +- `req~java-scriptclass-option-handling~1` + +### Java %jar Option Handling in Design +`dsn~java-jar-option-handling~1` + +Design the parser to collect and handle multiple %jar options. Ensure that they follow the Java CLASSPATH environment variable syntax (colon-separated values), remove duplicates, and maintain the order specified in the script by using sets and lists. + +Needs: req +Covers: +- `req~java-jar-option-handling~1` + +### Java %jvmoption Handling in Design +`dsn~java-jvmoption-handling~1` + +Create a way for the parser to collect multiple %jvmoption options, allowing duplicates and preserving the order specified. This can be achieved using a list to store the options as they are parsed. + +Needs: req +Covers: +- `req~java-jvmoption-handling~1` + +### Java %import Option Handling in Design +`dsn~java-import-option-handling~1` + +Implement logic to process %import options by interacting with the Swig Metadata object. The parser will replace the found %import Script Option with the referenced script code and handle nested %jar, %jvmoption, and %import options, ignoring %scriptclass options in imported scripts. + +Needs: req +Covers: +- `req~java-import-option-handling~1` + +### General Parser Integration +`dsn~general-parser-integration~1` + +Ensure that the new parser integrates seamlessly into the Exasol UDF Client environment. This includes embedding the parser within the custom C++ namespace, ensuring it meets all linker requirements, and does not introduce additional runtime dependencies. + +Needs: req +Covers: +- `req~new-parser-integration~1` + +## Architecture Overview + +This section provides a high-level overview of the system architecture for the UDF Client Script Options parser. + +### System Context + +#### External Interfaces +- **UDF Client**: Communicates with the parser to process script options within UDFs. +- **Exasol Database**: Provides the data environment where UDF scripts are executed. + +### Components + +#### Parser Component +- Implemented using the [ctpg](https://github.com/peter-winter/ctpg) parser library with Lexer and Parser rules defined in C++. +- Manages the identification, handling, and removal of Script Options, including Java-specific options like %scriptclass, %jar, %jvmoption, and %import. + +#### Integration Component +- Ensures the parser is embedded within the UDF Client namespace without runtime dependencies. +- Handles interaction with the Swig Metadata object for processing %import options. + +## Detailed Design + +### Parser Component Design + +1. **Lexer Rules** + - Define tokens for `%optionKey`, `optionValue`, and whitespace characters including `\t`, `\v`, `\f`. + +2. **Parser Rules** + - Define the grammar for identifying and processing Script Options. + - Handle multiple options with the same key and manage duplicates as specified. + - Implement rules to replace escape sequences (e.g., `\\`, `\n`, `\r`) in option values. + - Ensure leading white spaces in option values are interpreted correctly. + +3. **Option Handling** + - Define a map of lists to collect and store Script Options by their keys. + - Implement logic to detect and discard additional %scriptclass options, maintaining only the first instance. + - Ensure %jar options follow the syntax of Java CLASSPATH and remove duplicates while preserving order using a set/list combination. + +### Integration Component Design + +1. **Namespace Embedding** + - Ensure the parser is embedded within the custom C++ namespace of the UDF Client to meet linker requirements. + +2. **Swig Metadata Interaction** + - Implement methods to interact with the Swig Metadata object for processing %import options. + - Replace found %import options with the referenced script code and manage any nested Script Options within the imported code, ignoring %scriptclass options. + +## Conclusion + +This design document outlines the detailed implementation for the new Exasol UDF Client Script Options parser, ensuring it meets all high-level and design requirements, integrates seamlessly into the existing environment, and provides robust handling of all specified Script Options. diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md new file mode 100644 index 00000000..5d94e1b2 --- /dev/null +++ b/exaudfclient/docs/script_options_requirments.md @@ -0,0 +1,124 @@ +# System Requirement Specification + +This document outlines the user-centric requirements for the new Exasol UDF Client Script Options parser. It identifies the key features, high-level requirements, and user roles to ensure clarity and alignment with user needs. + +## Roles + +This section lists the roles that will interact with or benefit from the parser system. + +### Database Administrator +Database Administrators manage the database environment and ensure the efficient execution of UDFs within Exasol, configuring and overseeing script execution. + +### Data Scientist +Data Scientists develop and deploy UDFs in languages such as Java, Python, or R to process and analyze data within Exasol. + +## Features + +This section lists the key features of the new UDF Client Script Options parser which you would highlight in a product leaflet. + +### General Script Options Parsing +`feat~general-script-options-parsing~1` + +Needs: req + +### Java-specific Script Options +`feat~java-specific-script-options~1` + +Needs: req + +## High-level Requirements + +This section details the high-level requirements for the new parser system, linked to the features listed above. + +### General Script Options Parsing +`req~general-script-options-parsing~1` + +The parser must correctly identify and handle Script Options with the syntax `%optionKey optionValue;`. It must also manage whitespace and ignore non-Script Options as specified. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` + +### Multiple and Duplicate Options Management +`req~multiple-duplicate-options-management~1` + +The parser must collect multiple Script Options with the same key and handle duplicates according to specific rules for different options. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` +- `feat~java-specific-script-options~1` + +### Script Option Removal +`req~script-option-removal~1` + +The parser must remove found Script Options from the original script code. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` +- `feat~java-specific-script-options~1` + +### Java %scriptclass Option Handling +`req~java-scriptclass-option-handling~1` + +The parser must correctly identify a single %scriptclass option and remove any additional occurrences of this option within the script code. + +Needs: dsn + +Covers: +- `feat~java-specific-script-options~1` + +### Java %jar Option Handling +`req~java-jar-option-handling~1` + +The parser must find multiple %jar options, handle duplicates properly, and ensure the values match the Java CLASSPATH environment variable syntax. + +Needs: dsn + +Covers: +- `feat~java-specific-script-options~1` + +### Java %jvmoption Handling +`req~java-jvmoption-handling~1` + +The parser must find multiple %jvmoption options, allowing duplicates and maintaining order. + +Needs: dsn + +Covers: +- `feat~java-specific-script-options~1` + +### Java %import Option Handling +`req~java-import-option-handling~1` + +For each found %import option, the parser must request and replace the found Script Option with the referenced script code, also handling nested options appropriately. + +Needs: dsn + +Covers: +- `feat~java-specific-script-options~1` + +### New Parser Implementation +`req~new-parser-implementation~1` + +The new parser must be implemented using an existing, open-source parser that supports definition of Lexer and Parser Rules in C++ code without additional runtime dependencies. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` + +### New Parser Integration +`req~new-parser-integration~1` + +Ensure that the new parser integrates seamlessly into the Exasol UDF Client environment, meeting all specified dependency and embedding requirements. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` +- `feat~java-specific-script-options~1` From 838396e670c06800bab17104ab5ac75b2c783ed8 Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Thu, 24 Oct 2024 09:31:52 -0300 Subject: [PATCH 02/16] Updated design document --- exaudfclient/docs/script_options_design.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/exaudfclient/docs/script_options_design.md b/exaudfclient/docs/script_options_design.md index f45687a0..27f57c19 100644 --- a/exaudfclient/docs/script_options_design.md +++ b/exaudfclient/docs/script_options_design.md @@ -13,12 +13,12 @@ The purpose of this document is to provide an in-depth implementation view of th ## Design Requirements -This section outlines the specific design requirements linked to the high-level requirements, detailing the solutions and methods to achieve them. +This section outlines the specific design requirements linked to the high-level requirements, detailing the implementation and methods to achieve them. ### Parser Implementation `dsn~parser-implementation~1` -Implementation of the parser using [ctpg](https://github.com/peter-winter/ctpg), an open-source parser library. This library will be used to define Lexer and Parser Rules in C++ code, ensuring no additional runtime dependencies exist. +Implement the parser using [ctpg](https://github.com/peter-winter/ctpg), an open-source parser library. This library will be used to define Lexer and Parser Rules in C++ code, ensuring no additional runtime dependencies exist. Needs: req Covers: @@ -118,7 +118,7 @@ This section provides a high-level overview of the system architecture for the U ## Detailed Design -### Parser Component Design +### Parser Component Implementation 1. **Lexer Rules** - Define tokens for `%optionKey`, `optionValue`, and whitespace characters including `\t`, `\v`, `\f`. @@ -134,15 +134,11 @@ This section provides a high-level overview of the system architecture for the U - Implement logic to detect and discard additional %scriptclass options, maintaining only the first instance. - Ensure %jar options follow the syntax of Java CLASSPATH and remove duplicates while preserving order using a set/list combination. -### Integration Component Design +### Integration Component Implementation 1. **Namespace Embedding** - Ensure the parser is embedded within the custom C++ namespace of the UDF Client to meet linker requirements. 2. **Swig Metadata Interaction** - Implement methods to interact with the Swig Metadata object for processing %import options. - - Replace found %import options with the referenced script code and manage any nested Script Options within the imported code, ignoring %scriptclass options. - -## Conclusion - -This design document outlines the detailed implementation for the new Exasol UDF Client Script Options parser, ensuring it meets all high-level and design requirements, integrates seamlessly into the existing environment, and provides robust handling of all specified Script Options. + - Replace found %import options with the referenced script code and manage any nested Script Options within the imported \ No newline at end of file From 2630c8035311fdd5497f3bfe5e32085f752e420f Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Thu, 24 Oct 2024 12:37:31 -0300 Subject: [PATCH 03/16] Findings review --- .../docs/script_options_requirments.md | 56 ++++++++++++++----- 1 file changed, 43 insertions(+), 13 deletions(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index 5d94e1b2..caa68001 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -19,11 +19,16 @@ This section lists the key features of the new UDF Client Script Options parser ### General Script Options Parsing `feat~general-script-options-parsing~1` +Script Options must be parsed according to syntax definition. +Developers can add additional Options in an easy and consistent way. + Needs: req ### Java-specific Script Options `feat~java-specific-script-options~1` +The parser must process all Java specific options correctly. + Needs: req ## High-level Requirements @@ -33,23 +38,32 @@ This section details the high-level requirements for the new parser system, link ### General Script Options Parsing `req~general-script-options-parsing~1` -The parser must correctly identify and handle Script Options with the syntax `%optionKey optionValue;`. It must also manage whitespace and ignore non-Script Options as specified. +The parser must correctly identify and handle Script Options with the syntax `%;`. It must also manage white spaces and ignore non-Script Options as specified. Needs: dsn Covers: - `feat~general-script-options-parsing~1` -### Multiple and Duplicate Options Management -`req~multiple-duplicate-options-management~1` +### Multiple and Options +`req~multiple-options-management~1` -The parser must collect multiple Script Options with the same key and handle duplicates according to specific rules for different options. +The parser must collect multiple Script Options with the same key. Note: the specific handling depends on the option handler. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` + +### Duplicate Options Management +`req~multiple-options-management~1` + +The parser must collect multiple Script Options with the same key and value. Note: the specific handling depends on the option handler. Needs: dsn Covers: - `feat~general-script-options-parsing~1` -- `feat~java-specific-script-options~1` ### Script Option Removal `req~script-option-removal~1` @@ -65,7 +79,7 @@ Covers: ### Java %scriptclass Option Handling `req~java-scriptclass-option-handling~1` -The parser must correctly identify a single %scriptclass option and remove any additional occurrences of this option within the script code. +The parser must correctly identify the first `%scriptclass` option and remove any additional occurrences of this option within the script code. Needs: dsn @@ -92,33 +106,49 @@ Needs: dsn Covers: - `feat~java-specific-script-options~1` +### Java %import Option Handling +`req~java-import-option-replace-referenced-scripts~1` + +For each found %import option, the parser must request and replace the referenced scripts recursively. This means, +if the referenced scripts contain also `%import` options, the implementation must replace those, too. + +Needs: dsn + +Covers: +- `feat~java-specific-script-options~1` + ### Java %import Option Handling `req~java-import-option-handling~1` -For each found %import option, the parser must request and replace the found Script Option with the referenced script code, also handling nested options appropriately. +For each found %import option, the parser must handle nested Script Options appropriately: +1. `%scriptclass` option must be ignored, but removed from the script code. +2. All other options must be handled as if they were part of the source script. Needs: dsn Covers: - `feat~java-specific-script-options~1` -### New Parser Implementation -`req~new-parser-implementation~1` +### Existing Parser Library Dependencies +`req~existing-parser-library-dependencies~1` The new parser must be implemented using an existing, open-source parser that supports definition of Lexer and Parser Rules in C++ code without additional runtime dependencies. +The implementation needs to be Open Source because the projects where the parser will be used are mainly Open Source, too. +It is important to avoid additional runtime dependencies, as this would complicate the setup and maintenance of the runtime environment of the UDF client (aka Script Languages Container). + Needs: dsn Covers: - `feat~general-script-options-parsing~1` -### New Parser Integration -`req~new-parser-integration~1` +### Existing Parser Linker Namespace Compatibility +`req~existing-parser-linker-namespace-compatibility~1` -Ensure that the new parser integrates seamlessly into the Exasol UDF Client environment, meeting all specified dependency and embedding requirements. +Ideally, the new parser should allow encapsulation in a custom C++ namespace, in order to avoid possible linker namespace conflicts with customer libraries. Needs: dsn Covers: - `feat~general-script-options-parsing~1` -- `feat~java-specific-script-options~1` + From 10c16c860b5597d9ee5dc7a871fee11355a9ae3b Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Thu, 24 Oct 2024 14:13:23 -0300 Subject: [PATCH 04/16] Rework --- .../docs/script_options_requirments.md | 181 ++++++++++++++++-- 1 file changed, 170 insertions(+), 11 deletions(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index caa68001..7101fcfe 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -38,13 +38,105 @@ This section details the high-level requirements for the new parser system, link ### General Script Options Parsing `req~general-script-options-parsing~1` -The parser must correctly identify and handle Script Options with the syntax `%;`. It must also manage white spaces and ignore non-Script Options as specified. +The parser must correctly identify and handle Script Options with the syntax `%;`. +The separator between Needs: dsn Covers: - `feat~general-script-options-parsing~1` +### White Spaces +`req~white-spaces~1` + +The following is the list of white spaces: +|======================================================= +| Name | C/Python/Java | ASCII Dec | ASCII Hex | +| tabulator | '\t' | 9 | 0x09 | +| vertical tab | '\v' | 11 | 0x0b | +| form feed | '\f' | 12 | 0x0c | +| space | ' ' | 30 | 0x20 | +|======================================================= + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` + + +### Leading White Spaces Options Parsing +`req~leading-white-spaces-script-options-parsing~1` + +The parser must recognize Script Options for lines starting with white space characters. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` + +Depends: + - `req~white-spaces~1` + +### Ignore anything which is not a Script Option +`req~ignore-none-script-options~1` + +If there is any character in front of a Script Option which is not a white space, the parser must ignore the option(s). + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` + +Depends: + - `req~white-spaces~1` + +### Multiple Line Script Options Parsing +`req~multiple-lines-script-options-parsing~1` + +The parser must recognize Script Options at any line in the given script code. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` + +### White Spaces Options Parsing V1 +`req~white-spaces-script-options-parsing-v1~1` + +All white spaces between the option key and option value are to be interpreted as separator. +White spaces between the option value and the terminating ";" are to be removed from the option value. + +Needs: dsn + +Tags: V1 + +Covers: +- `feat~general-script-options-parsing~1` + +Depends: + - `req~white-spaces~1` + +### White Spaces Options Parsing V2 +`req~white-spaces-script-options-parsing-v2~1` + +All white spaces between the option key and option value are to be ignored. The following rules for escape sequences at **the start** of a script optionValue are to be applied: +- '\ ' => space character +- '\t' => character +- '\f' =>
character +- '\v' => character + +White spaces between the option value and the terminating ";" shall remain. + +Needs: dsn + +Tags: V2 + +Covers: +- `feat~general-script-options-parsing~1` + +Depends: + - `req~white-spaces~1` + ### Multiple and Options `req~multiple-options-management~1` @@ -68,38 +160,101 @@ Covers: ### Script Option Removal `req~script-option-removal~1` -The parser must remove found Script Options from the original script code. +The parser handler must remove found known Script Options from the original script code. Successfully parsed Script Options, which are not recognized by the parser handler shall remain in the script code. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` +- `feat~java-specific-script-options~1` + + +### Escape Sequence Script Options Parsing +`req~escape-sequence-script-options-parsing~1` + +The following rules for escape sequences at any place within a script optionValue are to be applied: +- '\n' => character +- '\r' => character +- '\;' => ';' character Needs: dsn +Tags: V2 + Covers: - `feat~general-script-options-parsing~1` + + +### Java %scriptclass Option Handling V1 +`req~java-scriptclass-option-handling-v1~1` + +The Java parser handler must correctly identify the first `%scriptclass` option and remove only this single instance from the script code. Any further occurrences of `%scriptclass` option shall stay in the source script code. + +Needs: dsn + +Tags: V1 + +Covers: +- `feat~java-specific-script-options~1` + + +### Java %scriptclass Option Handling V2 +`req~java-scriptclass-option-handling-v2~1` + +The Java parser handler must correctly identify the first `%scriptclass` option and remove any additional occurrences of this option within the script code. + +Needs: dsn + +Tags: V2 + +Covers: +- `feat~java-specific-script-options~1` + +### Java %jar Option Handling V1 +`req~java-jar-option-handling-v1~1` + +The Java parser handler must find multiple %jar options. The values are to be interpreted as the Java CLASSPATH: `::...:`. +The Java parser handler shall split the entries by the colon character. The Java parser handler shall identify duplicated files and order the result of all `%jar` options alphabetically. + +Needs: dsn + +Tags: V1 + +Covers: - `feat~java-specific-script-options~1` -### Java %scriptclass Option Handling -`req~java-scriptclass-option-handling~1` +### Java %jar Option Handling V2 +`req~java-jar-option-handling-v2~1` -The parser must correctly identify the first `%scriptclass` option and remove any additional occurrences of this option within the script code. +The Java parser handler must find multiple %jar options. The values are to be interpreted as the Java CLASSPATH: `::...:`. +The Java parser handler shall split the entries by the colon character. The Java parser handler must keep duplicates. The order of the entries must not change. Needs: dsn +Tags: V2 + Covers: - `feat~java-specific-script-options~1` -### Java %jar Option Handling -`req~java-jar-option-handling~1` +### Java %jar Option Trailing White Space Handling +`req~java-jar-option-trailing-white-space-handling~1` -The parser must find multiple %jar options, handle duplicates properly, and ensure the values match the Java CLASSPATH environment variable syntax. +The Java parser handler must remove trailing white spaces for `%jar` option values if they are part of the escape sequence '\ '. Escape sequences at the end of a found `%jar` option of the form `\ ` must be replaced with ' '. +This approach provides backwards compatibility for most existing UDF's from customers. Needs: dsn +Tags: V2 + Covers: - `feat~java-specific-script-options~1` +Depends: +- `req~white-spaces-script-options-parsing-v2~1` ### Java %jvmoption Handling `req~java-jvmoption-handling~1` -The parser must find multiple %jvmoption options, allowing duplicates and maintaining order. +The Java parser handler must find multiple %jvmoption options, allowing duplicates and maintaining order. Needs: dsn @@ -109,7 +264,7 @@ Covers: ### Java %import Option Handling `req~java-import-option-replace-referenced-scripts~1` -For each found %import option, the parser must request and replace the referenced scripts recursively. This means, +For each found %import option, the Java parser handler must request and replace the referenced scripts recursively. This means, if the referenced scripts contain also `%import` options, the implementation must replace those, too. Needs: dsn @@ -120,7 +275,7 @@ Covers: ### Java %import Option Handling `req~java-import-option-handling~1` -For each found %import option, the parser must handle nested Script Options appropriately: +For each found %import option, the Java parser handler must handle nested Script Options appropriately: 1. `%scriptclass` option must be ignored, but removed from the script code. 2. All other options must be handled as if they were part of the source script. @@ -139,6 +294,8 @@ It is important to avoid additional runtime dependencies, as this would complica Needs: dsn +Tags: V2 + Covers: - `feat~general-script-options-parsing~1` @@ -149,6 +306,8 @@ Ideally, the new parser should allow encapsulation in a custom C++ namespace, in Needs: dsn +Tags: V2 + Covers: - `feat~general-script-options-parsing~1` From 6de77fed77a681958f2c3893ed204707f586c0fa Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Fri, 25 Oct 2024 09:33:13 -0300 Subject: [PATCH 05/16] Improved requirements --- exaudfclient/docs/script_options_requirments.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index 7101fcfe..b1f2f653 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -189,6 +189,7 @@ Covers: `req~java-scriptclass-option-handling-v1~1` The Java parser handler must correctly identify the first `%scriptclass` option and remove only this single instance from the script code. Any further occurrences of `%scriptclass` option shall stay in the source script code. +The value should be handled according to the [Java specification for identifies](https://docs.oracle.com/javase/specs/jls/se7/html/jls-3.html#jls-3.8). Needs: dsn @@ -202,6 +203,7 @@ Covers: `req~java-scriptclass-option-handling-v2~1` The Java parser handler must correctly identify the first `%scriptclass` option and remove any additional occurrences of this option within the script code. +The value should be handled according to the [Java specification for identifies](https://docs.oracle.com/javase/specs/jls/se7/html/jls-3.html#jls-3.8). Needs: dsn @@ -251,6 +253,7 @@ Covers: Depends: - `req~white-spaces-script-options-parsing-v2~1` + ### Java %jvmoption Handling `req~java-jvmoption-handling~1` @@ -261,11 +264,12 @@ Needs: dsn Covers: - `feat~java-specific-script-options~1` -### Java %import Option Handling +### Java %import Option Replace Referenced Sripts `req~java-import-option-replace-referenced-scripts~1` For each found %import option, the Java parser handler must request and replace the referenced scripts recursively. This means, if the referenced scripts contain also `%import` options, the implementation must replace those, too. +The referenced script name should be handled according to the [Exasol SQL identifier specification](https://docs.exasol.com/db/latest/sql_references/basiclanguageelements.htm#SQLidentifier). Needs: dsn From 231bbd433901a189f78e2dd15609e3895ab1b2c4 Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Fri, 25 Oct 2024 10:13:35 -0300 Subject: [PATCH 06/16] Fixed requirment name --- exaudfclient/docs/script_options_requirments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index b1f2f653..2a61eb69 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -137,7 +137,7 @@ Covers: Depends: - `req~white-spaces~1` -### Multiple and Options +### Multiple Options `req~multiple-options-management~1` The parser must collect multiple Script Options with the same key. Note: the specific handling depends on the option handler. From 48c58b4b380ae876e74a8082692cce26855b9470 Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Fri, 25 Oct 2024 10:42:45 -0300 Subject: [PATCH 07/16] Removed design doc --- exaudfclient/docs/script_options_design.md | 144 --------------------- 1 file changed, 144 deletions(-) delete mode 100644 exaudfclient/docs/script_options_design.md diff --git a/exaudfclient/docs/script_options_design.md b/exaudfclient/docs/script_options_design.md deleted file mode 100644 index 27f57c19..00000000 --- a/exaudfclient/docs/script_options_design.md +++ /dev/null @@ -1,144 +0,0 @@ -# Design Document - -This document details the design aspects of the new Exasol UDF Client Script Options parser, based on the high-level requirements outlined in the System Requirement Specification. - -## Introduction - -The purpose of this document is to provide an in-depth implementation view of the UDF Client Script Options parser, ensuring it meets all specified high-level requirements and integrates seamlessly into the existing Exasol environment. The parser needs to be implemented in C++. - -## Constraints - -- The parser implementation must be in C++. -- The chosen parser implementation is [ctpg](https://github.com/peter-winter/ctpg), which supports the definition of Lexer and Parser Rules in C++ code. - -## Design Requirements - -This section outlines the specific design requirements linked to the high-level requirements, detailing the implementation and methods to achieve them. - -### Parser Implementation -`dsn~parser-implementation~1` - -Implement the parser using [ctpg](https://github.com/peter-winter/ctpg), an open-source parser library. This library will be used to define Lexer and Parser Rules in C++ code, ensuring no additional runtime dependencies exist. - -Needs: req -Covers: -- `req~new-parser-implementation~1` - -### Lexer and Parser Rules -`dsn~lexer-parser-rules~1` - -Define the Lexer rules to tokenize `%optionKey`, `optionValue`, and allowed whitespace characters, including `\t`, `\v`, and `\f`. The Parser rules will define the grammar to correctly identify Script Options, manage multiple options with the same key, and handle duplicates. - -Needs: req -Covers: -- `req~general-script-options-parsing~1` - -### Handling Multiple and Duplicate Options -`dsn~handling-multiple-duplicate-options~1` - -Create a mechanism within the parser to collect and aggregate multiple Script Options with the same key. Use a data structure (e.g., a map of lists) to store options by key and ensure duplicates are handled according to specified rules for different options. - -Needs: req -Covers: -- `req~multiple-duplicate-options-management~1` - -### Script Option Removal Mechanism -`dsn~script-option-removal-mechanism~1` - -Implement a method to remove identified Script Options from the original script code. This method will traverse the code, identify Script Options using defined tokens, and replace them with whitespace to ensure the cleaned script executes smoothly. - -Needs: req -Covers: -- `req~script-option-removal~1` - -### Java %scriptclass Option Handling in Design -`dsn~java-scriptclass-option-handling~1` - -Design logic to correctly identify a single %scriptclass option within the script. Additional occurrences of %scriptclass will be removed from the script code. Use a flag to mark the first occurrence and ensure subsequent ones are discarded. - -Needs: req -Covers: -- `req~java-scriptclass-option-handling~1` - -### Java %jar Option Handling in Design -`dsn~java-jar-option-handling~1` - -Design the parser to collect and handle multiple %jar options. Ensure that they follow the Java CLASSPATH environment variable syntax (colon-separated values), remove duplicates, and maintain the order specified in the script by using sets and lists. - -Needs: req -Covers: -- `req~java-jar-option-handling~1` - -### Java %jvmoption Handling in Design -`dsn~java-jvmoption-handling~1` - -Create a way for the parser to collect multiple %jvmoption options, allowing duplicates and preserving the order specified. This can be achieved using a list to store the options as they are parsed. - -Needs: req -Covers: -- `req~java-jvmoption-handling~1` - -### Java %import Option Handling in Design -`dsn~java-import-option-handling~1` - -Implement logic to process %import options by interacting with the Swig Metadata object. The parser will replace the found %import Script Option with the referenced script code and handle nested %jar, %jvmoption, and %import options, ignoring %scriptclass options in imported scripts. - -Needs: req -Covers: -- `req~java-import-option-handling~1` - -### General Parser Integration -`dsn~general-parser-integration~1` - -Ensure that the new parser integrates seamlessly into the Exasol UDF Client environment. This includes embedding the parser within the custom C++ namespace, ensuring it meets all linker requirements, and does not introduce additional runtime dependencies. - -Needs: req -Covers: -- `req~new-parser-integration~1` - -## Architecture Overview - -This section provides a high-level overview of the system architecture for the UDF Client Script Options parser. - -### System Context - -#### External Interfaces -- **UDF Client**: Communicates with the parser to process script options within UDFs. -- **Exasol Database**: Provides the data environment where UDF scripts are executed. - -### Components - -#### Parser Component -- Implemented using the [ctpg](https://github.com/peter-winter/ctpg) parser library with Lexer and Parser rules defined in C++. -- Manages the identification, handling, and removal of Script Options, including Java-specific options like %scriptclass, %jar, %jvmoption, and %import. - -#### Integration Component -- Ensures the parser is embedded within the UDF Client namespace without runtime dependencies. -- Handles interaction with the Swig Metadata object for processing %import options. - -## Detailed Design - -### Parser Component Implementation - -1. **Lexer Rules** - - Define tokens for `%optionKey`, `optionValue`, and whitespace characters including `\t`, `\v`, `\f`. - -2. **Parser Rules** - - Define the grammar for identifying and processing Script Options. - - Handle multiple options with the same key and manage duplicates as specified. - - Implement rules to replace escape sequences (e.g., `\\`, `\n`, `\r`) in option values. - - Ensure leading white spaces in option values are interpreted correctly. - -3. **Option Handling** - - Define a map of lists to collect and store Script Options by their keys. - - Implement logic to detect and discard additional %scriptclass options, maintaining only the first instance. - - Ensure %jar options follow the syntax of Java CLASSPATH and remove duplicates while preserving order using a set/list combination. - -### Integration Component Implementation - -1. **Namespace Embedding** - - Ensure the parser is embedded within the custom C++ namespace of the UDF Client to meet linker requirements. - -2. **Swig Metadata Interaction** - - Implement methods to interact with the Swig Metadata object for processing %import options. - - Replace found %import options with the referenced script code and manage any nested Script Options within the imported \ No newline at end of file From 94de611b1f155efb79d921751974b4382bdcdddf Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Mon, 28 Oct 2024 06:05:28 -0300 Subject: [PATCH 08/16] Fixes from review --- .../docs/script_options_requirments.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index 2a61eb69..1e953724 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -39,7 +39,6 @@ This section details the high-level requirements for the new parser system, link `req~general-script-options-parsing~1` The parser must correctly identify and handle Script Options with the syntax `%;`. -The separator between Needs: dsn @@ -51,7 +50,7 @@ Covers: The following is the list of white spaces: |======================================================= -| Name | C/Python/Java | ASCII Dec | ASCII Hex | +| Name | C syntax | ASCII Dec | ASCII Hex | | tabulator | '\t' | 9 | 0x09 | | vertical tab | '\v' | 11 | 0x0b | | form feed | '\f' | 12 | 0x0c | @@ -64,10 +63,24 @@ Covers: - `feat~general-script-options-parsing~1` +### White Spaces Separator +`req~white-spaces-separator~1` + +A Script Options can have one or more white spaces between the Script Option Key `` and Script Option Value `<`. +The parser must recognize those white spaces as separator(s). + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` + +Depends: + - `req~white-spaces~1` + ### Leading White Spaces Options Parsing `req~leading-white-spaces-script-options-parsing~1` -The parser must recognize Script Options for lines starting with white space characters. +The parser must recognize Script Options for lines starting with white space characters before the Script Options. Needs: dsn From d9caa54393923dec98901ad12e802cd7a61a5ff2 Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:20:17 -0300 Subject: [PATCH 09/16] Fixes from review --- .../docs/script_options_requirments.md | 47 ++++++++++++++----- 1 file changed, 35 insertions(+), 12 deletions(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index 1e953724..6890c92e 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -19,7 +19,7 @@ This section lists the key features of the new UDF Client Script Options parser ### General Script Options Parsing `feat~general-script-options-parsing~1` -Script Options must be parsed according to syntax definition. +Script Options must be parsed according to a given syntax definition. Developers can add additional Options in an easy and consistent way. Needs: req @@ -27,7 +27,7 @@ Needs: req ### Java-specific Script Options `feat~java-specific-script-options~1` -The parser must process all Java specific options correctly. +The parser must process all Java-specific options correctly. Needs: req @@ -48,7 +48,7 @@ Covers: ### White Spaces `req~white-spaces~1` -The following is the list of white spaces: +The parser must treat the following list of white spaces as token separator: |======================================================= | Name | C syntax | ASCII Dec | ASCII Hex | | tabulator | '\t' | 9 | 0x09 | @@ -80,7 +80,7 @@ Depends: ### Leading White Spaces Options Parsing `req~leading-white-spaces-script-options-parsing~1` -The parser must recognize Script Options for lines starting with white space characters before the Script Options. +The parser must accept Script Options for lines starting with any number of white space characters before the Script Options. Needs: dsn @@ -225,11 +225,19 @@ Tags: V2 Covers: - `feat~java-specific-script-options~1` +### Java %jar Option Handling Multiple Options +`req~java-jar-option-handling-multiple-options~1` + +The Java parser handler must find multiple %jar options. The values are to be interpreted as the Java CLASSPATH: `::...:`. +The Java parser handler shall split the entries by the colon character. + +Covers: +- `feat~java-specific-script-options~1` + ### Java %jar Option Handling V1 `req~java-jar-option-handling-v1~1` -The Java parser handler must find multiple %jar options. The values are to be interpreted as the Java CLASSPATH: `::...:`. -The Java parser handler shall split the entries by the colon character. The Java parser handler shall identify duplicated files and order the result of all `%jar` options alphabetically. +The Java parser handler shall identify duplicated files and order the result of all `%jar` options alphabetically. Needs: dsn @@ -238,11 +246,13 @@ Tags: V1 Covers: - `feat~java-specific-script-options~1` +Depends: +- `req~java-jar-option-handling-multiple-options~1` + ### Java %jar Option Handling V2 `req~java-jar-option-handling-v2~1` -The Java parser handler must find multiple %jar options. The values are to be interpreted as the Java CLASSPATH: `::...:`. -The Java parser handler shall split the entries by the colon character. The Java parser handler must keep duplicates. The order of the entries must not change. +The Java parser handler must keep duplicates. The order of the entries must not change. Needs: dsn @@ -251,10 +261,13 @@ Tags: V2 Covers: - `feat~java-specific-script-options~1` +Depends: +- `req~java-jar-option-handling-multiple-options~1` + ### Java %jar Option Trailing White Space Handling `req~java-jar-option-trailing-white-space-handling~1` -The Java parser handler must remove trailing white spaces for `%jar` option values if they are part of the escape sequence '\ '. Escape sequences at the end of a found `%jar` option of the form `\ ` must be replaced with ' '. +The Java parser handler must remove trailing white spaces for `%jar` option values if they are not part of the escape sequence '\ '. Escape sequences at the end of a found `%jar` option of the form `\ ` must be replaced with ' '. This approach provides backwards compatibility for most existing UDF's from customers. Needs: dsn @@ -280,8 +293,16 @@ Covers: ### Java %import Option Replace Referenced Sripts `req~java-import-option-replace-referenced-scripts~1` -For each found %import option, the Java parser handler must request and replace the referenced scripts recursively. This means, -if the referenced scripts contain also `%import` options, the implementation must replace those, too. +For each found `%import` option, the Java parser handler must request and replace the referenced scripts recursively. This means, if the referenced scripts contain also `%import` options, the implementation must replace those, too. + +Needs: dsn + +Covers: +- `feat~java-specific-script-options~1` + +### Java %import Option Referenced Sripts Name +`req~java-import-option-referenced-scripts-name~1` + The referenced script name should be handled according to the [Exasol SQL identifier specification](https://docs.exasol.com/db/latest/sql_references/basiclanguageelements.htm#SQLidentifier). Needs: dsn @@ -292,9 +313,10 @@ Covers: ### Java %import Option Handling `req~java-import-option-handling~1` -For each found %import option, the Java parser handler must handle nested Script Options appropriately: +For each found `%import` option, the Java parser handler must handle nested Script Options appropriately: 1. `%scriptclass` option must be ignored, but removed from the script code. 2. All other options must be handled as if they were part of the source script. +3. Already imported scripts must not be imported again, but the `%import` statement must be removed Needs: dsn @@ -307,6 +329,7 @@ Covers: The new parser must be implemented using an existing, open-source parser that supports definition of Lexer and Parser Rules in C++ code without additional runtime dependencies. The implementation needs to be Open Source because the projects where the parser will be used are mainly Open Source, too. It is important to avoid additional runtime dependencies, as this would complicate the setup and maintenance of the runtime environment of the UDF client (aka Script Languages Container). +Also, the license of the library must allow usage in closed source, i.g. MIT License. Needs: dsn From f3a024ecf3fc63f62e0cfe36c98fbd346346f167 Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Thu, 7 Nov 2024 08:45:35 -0300 Subject: [PATCH 10/16] Fixes from review --- exaudfclient/docs/script_options_requirments.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index 6890c92e..5331d8e6 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -138,7 +138,7 @@ All white spaces between the option key and option value are to be ignored. The - '\f' => character - '\v' => character -White spaces between the option value and the terminating ";" shall remain. +White spaces between the option value and the terminating ";" shall be interpreted as part of the value. Needs: dsn @@ -153,7 +153,10 @@ Depends: ### Multiple Options `req~multiple-options-management~1` -The parser must collect multiple Script Options with the same key. Note: the specific handling depends on the option handler. +The parser must collect multiple Script Options with the same key. + +Comment: +"Collecting" in this context means that a merging strategy must be applied. We don't want multiple options with the same key to result in simply overwriting or discarding values. Please note that the specific handling depends on the individual option handler. Needs: dsn @@ -228,8 +231,7 @@ Covers: ### Java %jar Option Handling Multiple Options `req~java-jar-option-handling-multiple-options~1` -The Java parser handler must find multiple %jar options. The values are to be interpreted as the Java CLASSPATH: `::...:`. -The Java parser handler shall split the entries by the colon character. +The Java parser handler must find multiple `%jar` options. The values are to be interpreted as the Java CLASSPATH: `::...:`. The Java parser handler shall split the entries by the colon character. Covers: - `feat~java-specific-script-options~1` @@ -237,7 +239,7 @@ Covers: ### Java %jar Option Handling V1 `req~java-jar-option-handling-v1~1` -The Java parser handler shall identify duplicated files and order the result of all `%jar` options alphabetically. +The Java parser handler shall unify duplicated files and order the result of all `%jar` options alphabetically. Needs: dsn @@ -283,7 +285,7 @@ Depends: ### Java %jvmoption Handling `req~java-jvmoption-handling~1` -The Java parser handler must find multiple %jvmoption options, allowing duplicates and maintaining order. +The Java parser handler must find multiple `%jvmoption` options, allowing duplicates and maintaining order. Needs: dsn From 7b0ff701b9aeac4ad35e4c0ba0b0f273dfb48d01 Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Thu, 7 Nov 2024 12:19:55 -0300 Subject: [PATCH 11/16] Fixes from review --- .../docs/script_options_requirments.md | 39 ++++++++++++++----- 1 file changed, 30 insertions(+), 9 deletions(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index 5331d8e6..7ba5ff62 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -166,7 +166,7 @@ Covers: ### Duplicate Options Management `req~multiple-options-management~1` -The parser must collect multiple Script Options with the same key and value. Note: the specific handling depends on the option handler. +The parser must collect multiple Script Options with the same key and same/different value. Note: the specific handling depends on the option handler. Needs: dsn @@ -176,7 +176,7 @@ Covers: ### Script Option Removal `req~script-option-removal~1` -The parser handler must remove found known Script Options from the original script code. Successfully parsed Script Options, which are not recognized by the parser handler shall remain in the script code. +The parser handler must remove found known Script Options from the original script code. Needs: dsn @@ -184,6 +184,31 @@ Covers: - `feat~general-script-options-parsing~1` - `feat~java-specific-script-options~1` +### Script Option Unknown Options Behavior V1 +`req~script-option-unknown-options-behvaior-v1~1` + +Unknown script options must remain untouched in the script code. The Java compiler is supposed to throw an error message during the compilation. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` +- `feat~java-specific-script-options~1` + +Tags: V1 + +### Script Option Unknown Options Behavior V2 +`req~script-option-unknown-options-behvaior-v2~1` + +The parser handler must detect unknown script options and throw an exception if such an options is found. + +Needs: dsn + +Covers: +- `feat~general-script-options-parsing~1` +- `feat~java-specific-script-options~1` + +Tags: V2 ### Escape Sequence Script Options Parsing `req~escape-sequence-script-options-parsing~1` @@ -325,14 +350,10 @@ Needs: dsn Covers: - `feat~java-specific-script-options~1` -### Existing Parser Library Dependencies -`req~existing-parser-library-dependencies~1` - -The new parser must be implemented using an existing, open-source parser that supports definition of Lexer and Parser Rules in C++ code without additional runtime dependencies. -The implementation needs to be Open Source because the projects where the parser will be used are mainly Open Source, too. -It is important to avoid additional runtime dependencies, as this would complicate the setup and maintenance of the runtime environment of the UDF client (aka Script Languages Container). -Also, the license of the library must allow usage in closed source, i.g. MIT License. +### Existing Parser Library License +`req~existing-parser-library-license~1` +The parser needs to be usable in open source and closed source projects. Needs: dsn From 9a1a3026d9b7282f0a0c07797a8ac96eb578d761 Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Mon, 11 Nov 2024 13:44:43 -0300 Subject: [PATCH 12/16] Fixes from review --- exaudfclient/docs/script_options_requirments.md | 17 ++--------------- 1 file changed, 2 insertions(+), 15 deletions(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index 7ba5ff62..02468a07 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -63,20 +63,6 @@ Covers: - `feat~general-script-options-parsing~1` -### White Spaces Separator -`req~white-spaces-separator~1` - -A Script Options can have one or more white spaces between the Script Option Key `` and Script Option Value `<`. -The parser must recognize those white spaces as separator(s). - -Needs: dsn - -Covers: -- `feat~general-script-options-parsing~1` - -Depends: - - `req~white-spaces~1` - ### Leading White Spaces Options Parsing `req~leading-white-spaces-script-options-parsing~1` @@ -138,7 +124,7 @@ All white spaces between the option key and option value are to be ignored. The - '\f' => character - '\v' => character -White spaces between the option value and the terminating ";" shall be interpreted as part of the value. +White spaces in the middle of the option value and between the option value and the terminating ";" shall be interpreted as part of the value. Needs: dsn @@ -257,6 +243,7 @@ Covers: `req~java-jar-option-handling-multiple-options~1` The Java parser handler must find multiple `%jar` options. The values are to be interpreted as the Java CLASSPATH: `::...:`. The Java parser handler shall split the entries by the colon character. +Compare [OpenJdk implementation](https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/java.base/share/classes/jdk/internal/loader/URLClassPath.java#L174) of parsing the classpath. Covers: - `feat~java-specific-script-options~1` From 7358ca6039c77f842f87b8b9726e25c5c388cf9a Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Tue, 12 Nov 2024 07:38:43 -0300 Subject: [PATCH 13/16] Fixes from review --- .../docs/script_options_requirments.md | 27 +++++++++++-------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index 02468a07..86d186f2 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -199,7 +199,7 @@ Tags: V2 ### Escape Sequence Script Options Parsing `req~escape-sequence-script-options-parsing~1` -The following rules for escape sequences at any place within a script optionValue are to be applied: +The following rules for escape sequences at any place within a script option value are to be applied: - '\n' => character - '\r' => character - '\;' => ';' character @@ -324,35 +324,40 @@ Needs: dsn Covers: - `feat~java-specific-script-options~1` -### Java %import Option Handling -`req~java-import-option-handling~1` +### Java %import Option Handling V1 +`req~java-import-option-handling-v1~1` For each found `%import` option, the Java parser handler must handle nested Script Options appropriately: -1. `%scriptclass` option must be ignored, but removed from the script code. +1. `%scriptclass` option must be ignored. 2. All other options must be handled as if they were part of the source script. 3. Already imported scripts must not be imported again, but the `%import` statement must be removed Needs: dsn +Tags: V1 + Covers: - `feat~java-specific-script-options~1` -### Existing Parser Library License -`req~existing-parser-library-license~1` +### Java %import Option Handling V2 +`req~java-import-option-handling-v2~1` -The parser needs to be usable in open source and closed source projects. +For each found `%import` option, the Java parser handler must handle nested Script Options appropriately: +1. `%scriptclass` option must be ignored, but removed from the script code. +2. All other options must be handled as if they were part of the source script. +3. Already imported scripts must not be imported again, but the `%import` statement must be removed Needs: dsn Tags: V2 Covers: -- `feat~general-script-options-parsing~1` +- `feat~java-specific-script-options~1` -### Existing Parser Linker Namespace Compatibility -`req~existing-parser-linker-namespace-compatibility~1` +### Existing Parser Library License +`req~existing-parser-library-license~1` -Ideally, the new parser should allow encapsulation in a custom C++ namespace, in order to avoid possible linker namespace conflicts with customer libraries. +The parser needs to be usable in open source and closed source projects. Needs: dsn From 583abb6e7bd0e01bd49fd1e22e61c832d425732c Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Wed, 13 Nov 2024 08:25:41 -0300 Subject: [PATCH 14/16] Updated FLAVOR_DESCRIPTION.md --- exaudfclient/docs/script_options_requirments.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index 86d186f2..cb92e6b6 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -12,6 +12,9 @@ Database Administrators manage the database environment and ensure the efficient ### Data Scientist Data Scientists develop and deploy UDFs in languages such as Java, Python, or R to process and analyze data within Exasol. +### Developer +Developers who integrates the Script Options parser library into other software. + ## Features This section lists the key features of the new UDF Client Script Options parser which you would highlight in a product leaflet. @@ -92,7 +95,7 @@ Depends: ### Multiple Line Script Options Parsing `req~multiple-lines-script-options-parsing~1` -The parser must recognize Script Options at any line in the given script code. +The parser must recognize Script Options at any line in the given script code. This is especially important because the `%import` option (see requirement `req~java-import-option-replace-referenced-scripts~1`) might replace options with Java code in the final script code. Needs: dsn @@ -125,6 +128,7 @@ All white spaces between the option key and option value are to be ignored. The - '\v' => character White spaces in the middle of the option value and between the option value and the terminating ";" shall be interpreted as part of the value. +The rationale is that the new version of the parser should be as much as possible backwards compatible to V1, because it will simplify migration of existing UDF's. Needs: dsn From 7c74c20822ace0af0d9e9d3537800ae4e4a5b03d Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Wed, 13 Nov 2024 08:29:29 -0300 Subject: [PATCH 15/16] Updated `req~existing-parser-library-license~1` --- exaudfclient/docs/script_options_requirments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index cb92e6b6..fe8d6e87 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -361,7 +361,7 @@ Covers: ### Existing Parser Library License `req~existing-parser-library-license~1` -The parser needs to be usable in open source and closed source projects. +Developers must be able to use the parser in open source and closed source projects. Needs: dsn From 675c75d3a575038604f9739c8b24c98caad909aa Mon Sep 17 00:00:00 2001 From: Thomas Ubensee <34603111+tomuben@users.noreply.github.com> Date: Wed, 13 Nov 2024 09:03:33 -0300 Subject: [PATCH 16/16] Apply suggestions from code review Co-authored-by: Torsten Kilias --- exaudfclient/docs/script_options_requirments.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/exaudfclient/docs/script_options_requirments.md b/exaudfclient/docs/script_options_requirments.md index fe8d6e87..a4a59bc0 100644 --- a/exaudfclient/docs/script_options_requirments.md +++ b/exaudfclient/docs/script_options_requirments.md @@ -95,7 +95,9 @@ Depends: ### Multiple Line Script Options Parsing `req~multiple-lines-script-options-parsing~1` -The parser must recognize Script Options at any line in the given script code. This is especially important because the `%import` option (see requirement `req~java-import-option-replace-referenced-scripts~1`) might replace options with Java code in the final script code. +The parser must recognize Script Options at any line in the given script code. + +Rationale: This is especially important because the `%import` option (see requirement `req~java-import-option-replace-referenced-scripts~1`) might replace options with Java code in the final script code. Needs: dsn @@ -121,14 +123,15 @@ Depends: ### White Spaces Options Parsing V2 `req~white-spaces-script-options-parsing-v2~1` -All white spaces between the option key and option value are to be ignored. The following rules for escape sequences at **the start** of a script optionValue are to be applied: +All white spaces between the option key and option value are to be ignored. The following rules for escape sequences at **the start** of a script option value are to be applied: - '\ ' => space character - '\t' => character - '\f' => character - '\v' => character White spaces in the middle of the option value and between the option value and the terminating ";" shall be interpreted as part of the value. -The rationale is that the new version of the parser should be as much as possible backwards compatible to V1, because it will simplify migration of existing UDF's. + +Rationale: The new version of the parser should be as much as possible backwards compatible to V1, because it will simplify migration of existing UDF's. Needs: dsn