Releases: cdapio/cdap
Releases · cdapio/cdap
Cask Data Application Platform v3.3.3
Bug Fix
- Fixes an issue that prevents MapReduce from running on clusters with HDFS encryption. (CDAP-4967)
Cask Data Application Platform v3.3.2
Bug Fix
- Fixed a schema-parsing bug that prevented the use of schemas where a record is used both as a top-level field and also used inside a different record field. (CDAP-4967)
- Worked around two issues (SPARK-13441 and YARN-4727) that prevented launching Spark jobs on Cloudera Data Hub clusters managed with Cloudera Manager when using Spark 1.4 or greater. (CDAP-5019)
- Fixed a problem with the CDAP Master not starting when CDAP and the HiveServer2 services are running on the same node in an Ambari cluster. (CDAP-5063)
- Fixed a problem with the CDAP CLI command "update app" that was parsing the application config incorrectly. (CDAP-5076)
- Fixed a problem where the explore schema fileset property was being ignored unless an explore format was also present. (CDAP-5094)
- Fix a problem with Spark jobs not being submitted to the appropriate YARN scheduler queue set for the namespace. (CDAP-5137)
Improvements
- Added a Batch Source Plugin to read from Microsoft Azure Blob Storage. (CDAP-5047)
- Added support for CDH 5.6 to CDAP. (CDAP-5134)
Cask Data Application Platform v3.3.1
Improvements
- Updated CDAP to use Tephra 0.6.5. (CDAP-4602)
- Added system metadata to existing entities. (CDAP-4708)
- Improved the Hydrator plugin archetypes to include build steps to build the deployment JSON for the artifact. (CDAP-4723)
- Improved the error logging for the Master Stream service when it can't connect to the CDAP AppFabric server. (CDAP-4773)
Bug Fixes
- Fixed an issue of not using the correct user to access HDFS when submitting a YARN application through Apache Twill, which caused cleanup failure on application termination. (CDAP-4117)
- Fixed a problem with tooltips not appearing in Flow and Workflow diagrams displayed in the Firefox browser. (CDAP-4613)
- The Hydrator UI now prevents drafts from being created with a name of an already-existing draft. This prevents overwriting of existing drafts. (CDAP-4679)
- Improved the metadata search to return matching entities from both the specified namespace and the system namespace. (CDAP-4688)
- Fixed a problem when using an Hbase sink as one of multiple sinks in a Hydrator pipeline. (CDAP-4689)
- Fixed an issue where system metadata updates were not being published to Kafka. (CDAP-4720)
- Fixed an issue where metadata updates wouldn't be sent when certain entities were deleted. (CDAP-4721)
- Added validation to the JSON imported in the Hydrator UI. (CDAP-4740)
- Fixed a bug with deleting artifact metadata when an artifact was deleted. (CDAP-4741)
- Fixed the Node.js server proxy to handle all backend errors (with and without statusCodes). (CDAP-4743)
- Fixed a bug in the Hydrator upgrade tool which caused drafts to not get upgraded. (CDAP-4745)
- Fixed the Hydrator Stream source to not assume an output schema. This is valid when a pipeline is created outside Hydrator UI. (CDAP-4753)
- Fixed ObjectStore to work when parameterized with custom classes. (CDAP-4754)
- Fixed an issue where delegation token cancellation of CDAP program was affecting CDAP master services. (CDAP-4767)
- Fixed the Cask Hydrator UI to automatically reconnect with the CDAP backend when the backend restarts. (CDAP-4770)
- Fixed an issue in Cloudera Manager installations where CDAP container logs would go to the stdout file instead of the master log. (CDAP-4771)
- Fixed an issue where the IndexedTable was dropping indices upon row updates. (CDAP-4784)
- Fixed a problem in the upgrade tool where deleted datasets would cause it to throw a NullPointerException. (CDAP-4785)
- Fixed an issue where the Hbase implementation of the Table API returned all rows, when the correct response should have been an empty set of columns. (CDAP-4790)
- Fixed a problem with the error message returned when loading an artifact with an invalid range. (CDAP-4800)
- Fixed the PartitionedFileSet's DynamicPartitioner to work with Avro OutputFormats. (CDAP-4806)
- Fixed a Validator Transform function generator in the Hydrator UI. (CDAP-4829)
- Allows user-scoped plugins to surface the correct widget JSON in the Hydrator UI. (CDAP-4831)
- Added the ErrorDataset as an option on widget JSON in Hydrator plugins. (CDAP-4832)
- Fixed a spacing issue for metrics showing in Pipeline diagrams of the Hydrator UI. (CDAP-4836)
- Fixed issues with the Hydrator UI widgets for the Hydrator Kafka real-time source, JMS real-time source, and CloneRecord transform. (CDAP-4853)
- Enhanced the CDAP SDK to be able to publish metadata updates to an external Kafka, identified by the configuration property metadata.updates.kafka.broker.list. Publishing can be enabled by setting metadata.updates.publish.enabled to true. Updates are published to the Kafka topic identified by the property metadata.updates.kafka.topic. (CDAP-4865)
- Fixed errors in Cask Hydrator Plugins. Two plugin documents (core-plugins/docs/Database-batchsink.md and core-plugins/docs/Database-batchsource.md) were removed, as the plugins have been moved from core-plugins to database-plugins (to database-plugins/docs/Database-batchsink.md and database-plugins/docs/Database-batchsource.md). (CDAP-4877)
- Fixed an issue with upgrading HBase tables while using the CDAP Upgrade Tool. (CDAP-4889)
- Fixed an issue with CDAP coprocessors that caused HBase tables to be disabled after upgrading the cluster to a highly-available file system. (CDAP-4894)
- Fixed the CDAP Upgrade Tool to return a non-zero exit status upon error during upgrade. (CDAP-4906)
- Fixed a PermGen memory leak that occurred while deploying multiple applications with database plugins. (CDAP-4924)
- Fixed the CDAP Explore Service JDBC driver to do nothing instead of throwing an exception when a commit is called. (CDAP-4927)
- Added an
'enableAutoCommit'property to the Cask Hydrator database plugins to enable the use of JDBC drivers that, similar to the Hive JDBC driver, do not allow commits. (CDAP-4950) - Changed the upload timeout from the CDAP CLI from 15 seconds to unlimited. (CDAP-4951)
- Pass ResourceManager delegation tokens in the proper format in secure Hadoop HA clusters. (CDAP-4975)
Deprecated and Removed Features
- See the CDAP 3.3.1 Javadocs for a list of deprecated and removed APIs.
- The properties
router.ssl.webapp.bind.port,router.webapp.bind.port,router.webapp.enabledhave been deprecated and will be removed in a future version.
Cask Data Application Platform v2.8.3
New Features
- Updated the version of Tephra used in CDAP to v0.4.4. (CDAP-4181)
Bug Fixes
- Fixed an issue about stopping all dependent services when a service is stopped. (CDAP-3861)
- Fixed an issue where socket connections to the TransactionManager were not being closed. (CDAP-4067)
- Fixed an issue that caused worker threads to go into an infinite recursion when exceptions were being thrown in channel handlers. (CDAP-4092)
Improvements
Cask Data Application Platform v3.3.0
New Features
- Added on demand (dynamic) dataset instantiation through program runtime context. (CDAP-961)
- Added lookup capability in context that can be used in existing Script, ScriptFilter and Validator transforms. (CDAP-2303)
- Added an endpoint to get a count of active queries:
/v3/namespaces/<namespace-id>/data/explore/queries/count. (CDAP-3514) - Added experimental support for running ETL Batch applications on Spark. Introduced an 'engine' setting in the configuration that defaults to
'mapreduce', but can be set to'spark'. (CDAP-3857) - Added support to PartitionConsumer for concurrency, plus a limit and filter on read. (CDAP-3944)
- Added support for limiting the number of concurrent schedule runs. (CDAP-3945)
- Added Java-8 support for Script transforms. (CDAP-4016)
- Added RESTful APIs to start or stop multiple programs. (CDAP-4022)
- Added CLI commands to stop, start, restart, or get status of programs in an application. (CDAP-4023)
- Added support for ETL transforms written in Python. (CDAP-4043)
- Added a new Javascript transform that can emit records using an emitter. (CDAP-4128)
- Added the capability for MapReduce and Spark programs to localize additional resources during setup. (CDAP-4135)
- Added the ability to configure which artifact a Hydrator plugin should use. (CDAP-4228)
- Added DAGs to ETL pipelines, which will allow users to fork and merge. ETLConfig has been updated to allow representing a DAG. (CDAP-4230)
- Added AuthorizationPlugin, for pluggable authorization. (CDAP-4235)
- Added metadata support for stream views. (CDAP-4263)
- Added CLI support for metadata and lineage. (CDAP-4270)
- Added the ability to add metadata to artifacts. (CDAP-4280)
- Added RESTful APIs to set and get properties for an artifact. (CDAP-4289)
- Added support for automatically annotating CDAP entities with system metadata when they are created or updated. (CDAP-4264)
- Added an authorization plugin that uses a system dataset to manage ACLs. (CDAP-4285)
- Moved Hydrator plugins from the CDAP repository as cdap-etl-lib into its own repository. (CDAP-4403)
- Improved Metadata Indexing and Search to support searches on words in value and tags. (CDAP-4591)
- Schema fields are stored as Metadata and are searchable. (CDAP-4592)
- Added capability in CDAP UI to display system tags. (CDAP-4658)
Improvements
- Table datasets, and any other dataset that implements
RecordWritable<StructuredRecord>, can now be written to using Hive. (CDAP-3079) - The CDAP Router now has a configurable timeout for idle connections, with a default timeout of 15 seconds. (CDAP-3887)
- A new property master.collect.containers.log has been added to cdap-site.xml, which determines if container logs are streamed back to the cdap-master process log. (This has always been the default behavior). For MapR installations, this must be turned off (set to false). (CDAP-4045)
- Added ability to retrieve the live-info for the AppFabric system service. (CDAP-4133)
- Added a method to
ObjectMappedTableandObjectStoreto retrieve a specific number of splits between a start and end keys. (CDAP-4209) - Messages logged by Hydrator are now prefixed with the name of the stage that logged them. (CDAP-4233)
- Added support for CDH5.5 (CDAP-4301)
- Upgraded netty-http dependency in CDAP to 0.14.0. (CDAP-4392)
- Make
xmllintdependency optional and allow setting variables to skip configuration file parsing. (CDAP-4444) - Added a schema validation -- for sources, transforms, and sinks -- that will validate the pipeline stages schema during deployment, and report any issues. (CDAP-4453)
- CDAP Master service will now log important configuration settings on startup. (CDAP-4518)
- Added the config setting
master.startup.checks.enabledto control whether CDAP Master startup checks are run or not. (CDAP-4523) - Improved the installation experience by adding to the CDAP Master service checks of pre-requisites such as file system permissions, availability of components such as YARN and HBase, resource availability during startup, and to error out if any of the pre-requisites fail. (CDAP-4536)
- Added a config setting 'master.collect.app.containers.log' that can be set to 'false' to disable streaming of application logs back to the CDAP Master log. (CDAP-4548)
- Added an error message when a required field is not provided when configuring Hydrator pipeline. (CDAP-4598)
Bug Fixes
- Prefix start script functions with
'cdap'to prevent namespace collisions. (CDAP-1174) - Added a check to cause a DB (source or sink) pipeline to fail during deployment if the table (source or sink) was not found, or if an incorrect connection string was provided. (CDAP-2470)
- Fixed a bug where the TTL for datasets was incorrect; it was reduced by (a factor of 1000) after an upgrade. After running the upgrade tool, please make sure the TTL values of tables are as expected. (CDAP-3345)
- Fixed an issue where the failure of a program running in a workflow fork node was causing other programs in the same fork node to remain in the RUNNING state, even after the Workflow was completed. (CDAP-3542)
- Fixed test failures in the PurchaseHistory, StreamConversion, and WikipediaPipeline example apps included in the CDAP SDK. (CDAP-3694)
- Fixed a bug where certain MapReduce metrics were not being properly emitted when using multiple outputs. (CDAP-3742)
- Fixed a problem with DBSink column names not being used to filter input record fields before writing to a DBSink. (CDAP-3761)
- Added a fix for case sensitivity handling in DBSink. (CDAP-3807)
- Fixed an issue where the regex filter for S3 Batch Source wasn't getting applied correctly. (CDAP-3815)
- Fixed an issue about stopping all dependent services when a service is stopped. (CDAP-3861)
- Fixed a bug when querying for logs of deleted program runs. (CDAP-3900)
- Fixed a problem with dataset performance degradation because of making multiple remote calls for each "get dataset" request. (CDAP-3902)
- Fixed QueryClient to work against HTTPS. (CDAP-3924)
- Fixed an issue where a stream that has a view could not be deleted cleanly. (CDAP-4000)
- Fixed an issue where socket connections to the TransactionManager were not being closed. (CDAP-4067)
- Fixes an issue that causes worker threads to go into an infinite recursion while exceptions are being thrown in channel handlers. (CDAP-4092)
- Fixed a bug that prevented applications from using HBase directly. (CDAP-4112)
- Fixed a problem where when CDAP Master switched from active to standby, the programs that were running were marked as failed. (CDAP-4119)
- Fixed a problem in the CLI command used to load an artifact, where the wrong artifact name and version was used if the artifact name ends with a number. (CDAP-4240)
- Fixed a problem where plugins from another namespace were visible when creating an application using a system artifact. (CDAP-4294)
- Fixed a problem with the CLI attempting to connect to CDAP when the hostname and port were incorrect. (CDAP-4316)
- Improved error message when stream views were not found. (CDAP-4366)
- Fixed an issue where tags search were failing for certain tags. (CDAP-4393)
- Fixed node.js version checking for the
cdap.shscript in the CDAP SDK. ([CDAP-4141](https://issues.cask.co...
Cask Data Application Platform v3.2.2
Improvements
- Made the metadata system datasets upgradable. (CDAP-3644)
- Revised the installation documentation for Cloudera Manager and Apache Ambari. (CDAP-4093)
- Added ability to get the live-info for the CDAP AppFabric system service. (CDAP-4133)
Bug Fixes
- Fixed an issue where socket connections to the TransactionManager were not being closed. (CDAP-4067)
- Fixes an issue that causes worker threads to go into an infinite recursion while exceptions are being thrown in channel handlers. (CDAP-4092)
- Fixed a problem where, when the CDAP Master switched from active to standby, that any programs that were running were marked as failed. (CDAP-4119)
- Fixed a problem that caused a Workflow to fail if datasets were used in custom actions. (CDAP-4278)
- Fixed a problem that prevented MapReduce jobs from being run when the Resource Manager switches from active to standby in a Kerberos-enabled HA cluster. (CDAP-4373)
- Fixed a problem that prevented streams from being read in HA HDFS mode. (CDAP-4384)
Known Issues
- Metadata search for tags does not work if the search key is not the first tag in the list. (CDAP-4393)
- See also the Known Issues of version 3.2.0.
Cask Data Application Platform v3.0.6
Bug Fixes
- Fixed a problem with faulty and unused metrics regarding CDAP file resource usage. (CDAP-3563).
- Fixed a problem where a Hadoop filesystem object was being instantiated before the
Kerberos keytab login was completed, leading to CDAP processes failing after the initial
ticket expired (CDAP-3660). - Fixed a problem where not all dependent services were being stopped when a system service was stopped (CDAP-3861).
- Fixed a problem with dataset performance degradation due to the making of
multiple remote calls for each "get dataset" request (CDAP-3902). - Fixed a problem with not closing the socket connection to the transaction service correctly,
leading to a leakage of file descriptors (CDAP-4067). - Fixed a problem with the CDAP Router being tied up in an infinite recursion (CDAP-4092).
- Fixed problems with the installation documentation (CDAP-4093).
- Fixed an issue with the incorrect labelling, after failover, of programs that had been running prior to failover (CDAP-4119).
New Features
Cask Data Application Platform v3.2.1
New Features
- Added the ability for S3 batch sources and sinks to set additional file system properties. (CDAP-3951)
Improvements
- Added logging and metrics support for Script, ScriptFilter, and Validator transforms. (CDAP-3870)
- Improved artifact and application deployment failure handling. (CDAP-3939)
Bug Fixes
- Fixed a problem with the CDAP SDK unable to start on certain Windows machines by updating
the Hadoop native library in CDAP with a version that does not have a dependency on a
debug version of the Microsoftmsvcr100.dll. (CDAP-3342) - Fixed an issue where the regex filter for S3 batch sources wasn't being applied correctly. (CDAP-3815)
- Fixed snapshot sinks so that the data is explorable as a
PartitionedFileSet. (CDAP-3829) - Fixed snapshot sinks so that they can be read safely. (CDAP-3833)
- Fixed a compilation error in the Maven application archetype. (CDAP-3859)
- Fixed a bug where plugins, packaged in the same artifact as an application class, could not be used by that application class. (CDAP-3860)
- Updated the documentation to remove references to application templates and adaptors that were removed as of CDAP 3.2.0. (CDAP-3891)
- Fixed a problem with running certain examples on Linux systems by increasing the maximum
Java heap size of the Standalone SDK on Linux systems to 2048m. (CDAP-3949) - Fixed a missing dependency on
cdap-hbase-compat-1.1package in the CDAP Master package. (CDAP-3961)
Cask Data Application Platform v3.2.0
New Features
- Added support for HBase1.1.(CDAP-2556)
- Added a new API for creating an application from an artifact.(CDAP-2666)
- Added the ability to write to multiple outputs from a MapReduce job.(CDAP-2756)
- Added the ability to dynamically write to multiple partitions of a PartitionedFileSet dataset as the output of a MapReduce job.(CDAP-2757)
- Added a Stream and Dataset Widget to the CDAP-UI.(CDAP-3253)
- Added stream views, enabling reading from a single stream using various formats and schemas.(CDAP-3390)
- Added a Validator Transform that can be used to validate records based on a set of available validators and configured to write invalid records to an error dataset.(CDAP-3476)
- Added a service to manage the metadata of CDAP entities.(CDAP-3516)
- Added the publishing of metadata change notifications to Apache Kafka.(CDAP-3518)
- Added the ability to compute lineage of a CDAP dataset or stream in a given time window.(CDAP-3519)
- Added RESTful APIs for adding/retrieving/deleting of metadata for apps/programs/datasets/streams.(CDAP-3520)
- Added the ability to record a dataset or stream access by a CDAP program.(CDAP-3521)
- Added the capability to search CDAP entities based on their metadata.(CDAP-3522)
- Added RESTful APIs for searching CDAP entities based on business metadata.(CDAP-3523)
- Added a data store to manage business metadata of CDAP entities.(CDAP-3527)
- Added SSH port forwarding to the CDAP virtual machine.(CDAP-3549)
- Added a data store for recording data accesses by CDAP programs and computing lineage.(CDAP-3556)
- Added the ability to write to multiple sinks in ETL real-time and batch applications.(CDAP-3590)
- Added the ability for real-time ETL pipelines to write to multiple sinks.(CDAP-3591)
- Added the ability for batch ETL pipelines to write to multiple sinks.(CDAP-3592)
- For the CSV and TSV stream formats, a “mapping” setting can now be specified, mapping stream event columns to schema columns.(CDAP-3626)
- Added support for CDAP to work with HDP 2.3.(CDAP-3693)
Improvements
- Added documentation of the RESTful endpoint to retrieve the properties of a stream.(CDAP-1914)
- Added an interface to load a file into a stream from the CDAP-UI.(CDAP-2514)
- The CDAP-UI “Errors” pop-up in the main screen now displays the time and date for each error.(CDAP-2809)
- Updated the Cloudera Manager CSD to use support for logback.(CDAP-2872)
- Cleaned up the messages shown in the errors dropdown in the CDAP-UI.(CDAP-2950)
- Added a CDAP-CLI command to stop a workflow.(CDAP-3147)
- Added support for upgrading the Hadoop distribution or the HBase version that CDAP is running on.(CDAP-3179)
- Revised the documentation of the file cdap-default.xml, removed properties no longer in use, and corrected discrepancies between the documentation and the shipped XML file.(CDAP-3257)
- Improved the help provided in the CDAP-CLI for the setting of stream formats.(CDAP-3270)
- Upgraded netty-http version to 0.12.0.(CDAP-3275)
- Added a HTTP RESTful API to update the application configuration and artifact version.(CDAP-3282)
- Added a “clear” button in the CDAP-UI for cases where a user decides to not used a pre-populated schema.(CDAP-3332)
- Defined a directory structure to be used for predefined applications.(CDAP-3351)
- Added documentation in the source code on adding new commands and completers to the CDAP-CLI.(CDAP-3357)
- In the CDAP-UI, added visualization for Workflow tokens in Workflows.(CDAP-3393)
- HBaseQueueDebugger now shows the minimum queue event transaction write pointer both for each queue and for all queues.(CDAP-3419)
- Added an example cdap-env.sh to the shipped packages.(CDAP-3443)
- Added an example in the documentation explaining how to prune invalid transactions from the transaction manager.(CDAP-3464)
- Modified the CDAP upgrade tool to delete all adapters and the ETLBatch and ETLRealtime ApplicationTemplates.(CDAP-3490)
- Added the ability to persist the runtime arguments with which a program was run.(CDAP-3495)
- Added support for writing to Amazon S3 in Avro and Parquet formats from batch ETL applications.(CDAP-3550)
- Updated CDAP to use Tephra 0.6.2.(CDAP-3564)
- Updated the transaction debugger client to print checkpoint information.(CDAP-3610)
Bug Fixes
- Fixed an issue where failed dataset operations via Explore queries did not invalidate the associated transaction.(CDAP-1697)
- Fixed a problem where users got an incorrect message while creating a dataset in a non-existent namespace.(CDAP-1864)
- Fixed a problem with services returning the same message for all failures.(CDAP-1892)
- Fixed a problem where a dataset could be created in a non-existent namespace in standalone mode.(CDAP-1984)
- Fixed a problem with the CDAP-CLI creating file logs.(CDAP-2428)
- Fixed a problem with the CDAP-CLI not auto-completing when setting a stream format.(CDAP-2521)
- Fixed a problem with the CDAP-UI of buttons staying ‘in focus’ after clicking.(CDAP-2785)
- The CDAP-UI “Errors” pop-up in the main screen now displays the time and date for each error.(CDAP-2809)
- Fixed a problem with schedules not being deployed in suspended mode.(CDAP-2892)
- Fixed a problem where failure of a spark node would cause a workflow to restart indefinitely.(CDAP-3014)
- Fixed an issue with the CDAP standalone process periodically crashing with Out-of-Memory errors when writing to an Oracle table.(CDAP-3073)
- Fixed a problem with workflow runs not getting scheduled due to Quartz exceptions.(CDAP-3101)
- Fixed a problem with discrepancies between the documentation and the defaults actually used by CDAP.(CDAP-3121)
- Fixed a problem in the CDAP-UI with the clone button in an incorrect position when using Firefox.(CDAP-3200)
- Fixed a problem in the CDAP-UI with an incorrect tabbing order when using Firefox.(CDAP-3201)
- Fixed a problem when specifying the HBase version using the HBASE_VERSION environment variable.(CDAP-3219)
- Fixed a problem in the CDAP-UI error pop-ups not having a default focus on a button.(CDAP-3233)
- Fixed a problem in the CDAP-UI with the default schema shown for streams.(CDAP-3243)
- Fixed a problem in the CDAP-UI with scrolling on the namespaces dropdown on certain pages.(CDAP-3260)
- Fixed a problem on CDAP distributed mode with the serializing of the metadata artifact causing a stack overflow.(CDAP-3261)
- Fixed a problem in the CDAP-UI not warning users if they exit or close their browser without saving.(CDAP-3305)
- Fixed a problem in the CDAP-UI with refreshing always returning to the overview page.(CDAP-3313)
- Fixed a problem with the table batch source requiring a row key to be set.(CDAP-3326)
- Fixed a problem with the application deployment for apps that contain Spark.(CDAP-3343)
- Fixed a problem with the display of ETL application metrics in the CDAP-UI.(CDAP-3349)
- Fixed a problem in the CDAP examples with the use of a runtime argument, min.pages.threshold.([CDAP-3355](https://issues.cask.co/browse/CDAP-335...
Cask Data Application Platform v3.1.2
- Improve the UI performance when rendering flow diagrams with a large number of nodes. (CDAP-3530)
- Fixed a bug that prevents streams events that are already processed from being re-processed in flows. (CDAP-3458)
- Fixed a bug that prevented explore service working on clusters with secure hive 0.14 (CDAP-3452)
- Fixed the readless increment co-processor to handle multiple readless increment columns in the same row. (CDAP-3449)
- Fixed a problem with the logback-container.xml not being copied into master services. (CDAP-3362)