Standard-flavor for Exasol 7.0+, Improved Flavor Customization and Error Reporting

Summary

This release introduces the new standard-flavor for Exasol 7.0+. Additionally, it adds new packages to the standard-flavors, such as boto3, pysftp and paramiko and fixes several bugs for the UDFs, the script-language containers and exaslct. Furthermore, we added more details to the error reporting of the UDFs. Error messages now contain error codes and for Java and Python they include untruncated stacktraces and chained exceptions.

However, most importantly, this release greatly simplifies the customization of the script-language containers. Each flavor now provides package lists under the directory flavors/<flavor-name>/flavor_customization/packages. Additional packages entered into these lists get automatically installed when building the flavor. Furthermore, we provide detailed instructions on how to do other kinds of modifications to the container by changing the Dockerfile at flavors/<flavor-name>/flavor_customization/Dockerfile. Please find more information about this, in our Readme.

NOTE: We cannot upload the pre-packaged python3-ds-cuda-preview container to GitHub, because it is larger than GitHub's file size limit for release artifacts. For this reason, we provide it under the following link. You can find the checksum of this container in the release artifacts, if you want to check its consistency.

Bug Fixes

• UDFs:

  • script-languages-release@#67: Fix error in Python3 UDFs when emitting only a dataframe with one Timestamp column
  • script-languages-release@#48: Fix Python3 UDF bug during reset after get_dataframe reaches the end of the iterator
  • script-languages-release@#31: Java EMITS UDF Scripts cannot emit null

• Script-Language Containers:

  • script-languages@#133: Pin tensorflow_hub version for python3-ds flavors
  • script-languages-release@#23: Pin azure-storage to version 0.36.0 until it is clear how we handle the deprecation with version 0.37.0

• Exaslct:

  • script-languages@#109: Fix bug in exaslct upload command
  • script-languages@#77: Handle tar return code during export command to detect potentially failed exports
  • script-languages-release@#60: Handle multiple lines at once in LogHandlers in exaslct to fix some bugs on MacOSx (but MacOSx is still highly experimental)

Features / Enhancements

• UDFs:

  • script-languages-release@#49: Introduce error codes into UDFClient
  • script-languages-release@#50: Introduce error codes for Java, Python and R UDFs. Return untruncated stacktraces for Java and Python. Include causes into Java stacktraces.
  • script-languages-release@#54: Support for chained exception for Python 3 UDFs
  • script-languages-release@#40: Improve shutdown handling of the UDFClient to reduce the risk of orphaned exaudfclient processes in error cases

• Script-Language Containers:

  • script-languages-release@#47: Add seperate standard-flavor for Exasol 7.0 with pinned package versions
  • script-languages-release@#38: Cleanup standard-flavor for Exasol 6.1+ to reduce the container size
  • script-languages-release@#26: Add boto3 to standard flavor
  • script-languages-release@#25: Add pysftp and paramiko to standard flavor
  • script-languages-release@#32: Add exasol websocket-api for Python3 to standard flavor
  • script-languages-release@#57: Simplify the customization of flavors for users
  • script-languages-release@#23: Add script to install specific version of R packages
  • script-languages@#127: Add more robust package install scripts which support package versions
  • script-languages-release@#94: Exclude unnecessary directories from /var during export
  • script-languages-release@#93: Add lists for actual installed packages to the release build step of the flavors
  • script-languages-release@#84: Copy apt/dpkg package database into release image

• Exslsct:

  • script-languages-release@#42: Add a command to generate a language activation statement
  • script-languages-release@#95: Generate and store checksum of the exported container besides the container

Refactorings

• Exaslct:

  • script-languages-release@#46: Reorganize exaslct and extract the test environment to exasol/integration-test-docker-environment
  • script-languages@#108: Add a cleanup method to exaslct tasks and scripts to run exaslct tests
  • script-languages@#98: Pass the docker socket to the test container
  • script-languages@#142: Extract and refactor functionalities for performance testing into exatest
  • script-languages@#105: Fix a reference to luigis FrozenOrderedDict and pin dependency versions in pipfile

• Continuous Integration:

  • script-languages-release@#99: Add a build breaker to the CI which fails if the commit was not rebuild without cache
  • script-languages-release@#87: Add pull-request blocker which checks if the HEAD of submodules is at a ancestor of origin/master