Skip to content

Latest commit

 

History

History
125 lines (109 loc) · 13.5 KB

Luna_Streaming_2.7_Release_Notes.md

File metadata and controls

125 lines (109 loc) · 13.5 KB
Raw

Release notes for DataStax Luna Streaming Distribution

Luna Streaming Distribution 2.7.2 is compatible with Apache Pulsar™ 2.7.2.

Release notes for Luna Streaming Distribution 2.7.2

11 June 2021

Component versions for Luna Streaming Distribution 2.7.2

  • Apache Pulsar 2.7.2
  • DataStax Pulsar Admin Console 1.0.0
  • DataStax Pulsar Heartbeat 1.0.2
  • DataStax Apache Pulsar Cassandra Sink 1.4.0
  • DataStax Apache Pulsar Cassandra Source 0.1.0
  • DataStax Burnell 1.0.0

This release adds these features to the original Apache Pulsar 2.7.2 release:

  • Stability improvements
  • Rootless Docker image
  • Schema API and Pulsar IO API improvements ported from Apache Pulsar 2.8.0
  • Dependency upgrades (for security, stability and performances)
  • An Enhanced ElasticSearch Pulsar IO Sink

Note: The DataStax Luna Streaming Distribution is designed for Java 11. However, because the product releases Docker images, you do not need to install Java (8 or 11) in advance. Java 11 is bundled in the Docker image.

Upgrade Considerations for Luna Streaming Distribution 2.7.2

This is the first Luna Streaming release that uses non-root containers for enhanced security. When upgrading from a previous version (for example, 2.6.2_1.0.1) files created while running that version will have root permissions and will not be readable by containers running the new version.

To fix this, you can manually log into the ZooKeeper, BookKeeper, and Function Worker containers and make sure that all files in the /pulsar/data/ and pulsar/logs directories are owned by UID 10000 (user pulsar). The group ID of the files should also be set to GID 10001 (group pulsar). Here is an example command:

chown -R 10000:10001 /pulsar/data

If you are are using the Luna Streaming Helm chart, you can enable automatic repair of the permissions using the fixRootlessPermissions setting. For more details on this setting, go here.

Luna Streaming Distribution 2.7.2 1.0.0

This release is based on the Apache Pulsar 2.7.2 release. In addition to the contents of that release, it includes the following notable commits:

  • 24787c7902d function container run as group 10001 mount function user token from k8s secret as 400 permission for rootless container
  • b3271b3ef34 Add workaround for failing PulsarFunctionsJavaProcessTest on JDK11 (#10566)
  • a742d138b46 PulsarFunctionsTests - use InstanceUtils.getDefaultSubscriptionName instead of an hardcoded subscription name
  • 96afcf94d5d Docker image: add vim and nettools (netstat)
  • 1d34fa639b1 Revert "[Schema] Support consume multiple schema types messages by AutoConsumeSchema (#10604)"
  • bfb25df7a14 Fix compile issue in AutoConsumeSchemaTest.java
  • 432f0960ff4 [ML] Cancel scheduled tasks as the first step in closing (#10739)
  • cb0dfd7cc6d [ML] Fix solution for preventing race conditions between timeout and completion (#10740)
  • 063040dfd38 Release OpAddEntry.data when entry is copied and discarded (#10773)
  • 9f30951139b [Broker] Fix possible data race in getFirstAvailableConsumerPermits (#10831)
  • 58a19fd6ce6 Fix the out of index issue when dispatch messages based on the avgBatchSizePerMsg. (#10828)
  • f508ff14dd2 Fix consumer stuck issue due to reuse entry wrapper. (#10824)
  • 7513f3e8e06 [Schema] Fix AutoConsumeSchema decode null schema version data (#10811)
  • e21c03884c1 AutoConsumeSchema: use decode(payload, schemaversion) (#10700)
  • 3398e3f74dc [Schema] Support consume multiple schema types messages by AutoConsumeSchema (#10604)
  • 66deda48eb8 ReflectionUtils use Class.forName in order to properly discover classes in Functions Runtime while using DefaultImplementation
  • 0d530108233 Move full pulsar-client-original in Pulsar IO classpath, as in 2.8.0 - this change allows to use KeyValueSchema and RecordSchemaBuilder in Pulsar Sinks
  • 0c62e0b412a AutoConsumeSchema: use decode(payload, schemaversion) (#10700)
  • 2bde8b20e14 [Schema] Support consume multiple schema types messages by AutoConsumeSchema (#10604)
  • 305db0e4e55 Support writing general records to Pulsar sink (#9590)
  • b6977c06aee Fix usage of seekAsync in MessageImpl.hasMessageAvailable and flaky-test (#10190)
  • 1f222bc77a3 Support get topic applied policy for message TTL (#9225)
  • 5d252e4c4a4 Ensure read-lock is not continuously held on a section while iterating over concurrent maps (#9787)
  • bedc715392b Branch-2.7: getListOfNamespaces does not fail with non-existant tenant
  • 4b9d2eede3d [Performance] Use single instance of parser (#10664)
  • 66a7d55993e Fix ConcurrentOpenLongPairRangeSet remove all range (#10656)
  • 7dd67603d13 Ensure all the ReadHandle gets properly closed on cache invalidation (#10659)
  • 6e0a03243aa Add metrics for nonContiguousDeletedMessagesRange (#10638)
  • 2c27f2c0db8 [C++] Avoid sending flow requests with zero permits (#10506)
  • 3ebd2d8b3b0 [Issue-10611] consumer related topic stats only available while consumer or reader are connected
  • ed1f33e8304 More fixes about running tests on JDK11 (#9893)
  • 79ddf32144e make ledger rollover check task internal (#8946)
  • c979b25736d Fix writing/encoding of GenericJsonRecord (#9608)
  • 5d4ad98933a Upgrade to Apache Avro 1.10.2 (#9898)
  • f270964367f [Security] Upgrade junit version to 4.13.1 to resolve CVE-2020-15250 and fix test dependency leak (#10147)
  • 8d006468a9e [Schema] Add schemaType field in SchemaHash (#10573)
  • 545edcb7412 Enhanced ES-sink connector
  • 0f11747c8e5 [CLIENT] fixed NPE in GenericJsonRecord (#10482)
  • bf008053ef3 [Issue 8751] Update Dockerfile for Pulsar and Dashboard to Create and Use pulsar User (nonroot user) (#8796)
  • bec07ebaa02 Pulsar Admin: return a better error message
  • 0ad9b90a5aa Comment lo4j2 script in order to save resources in pulsar-admin (#9309)
  • 203b67b366f Ensure that temporary directories are deleted during tests (#9263)
  • 0b9509a6fe6 Changes required to backport PR 7266 to 2.7 branch
  • d6027a8ea58 [pulsar-broker] Dispatch batch messages according consumer permits (#7266)
  • bcb849e0a01 Fixed missed ZK caching when fetching list of namespaces for a tenant (#10594)
  • 6f6dc57c6f0 [Issue 10010][Client] fixed memory leak (#10028)
  • 5d7ef69f156 [pulsar-broker] Dispatch batch messages according consumer permits (#7266)
  • d5aeb665c9d Add AvroSchema UUID support
  • 1a94bbcdc7b [Build] Specify release in maven-compiler-plugin configuration on JDK11 (#10343)
  • 5dee3ebe51e Use Message.getReaderSchema() in Pulsar IO Sinks when possible (#10557)
  • 7e1f9f06f2a Allow to build Pulsar with JDK11 and -Dmaven.compiler.release=8 (#9580)
  • e7670be4055 Upgrade Lombok to 1.18.20 (#10259)
  • e2c52457d27 Made OpAddEntry.toString() more robust to nulls to prevent NPEs (#10548)
  • a26de54f1bf [pulsar-broker] Dispatch messaages to consumer with permits (#10417)
  • 83d10a140f2 [Broker] Make Persistent*DispatcherMultipleConsumers.readMoreEntries synchronized (#10413)
  • a1e1388111f [pulsar-broker] Allow broker to discover and unblock stuck subscription (#9789)
  • 7e57cbe5de2 Add streaming dispatcher. (#9056)
  • c73cdae681f AutoConsumeSchema: handle schema NONE as BYTES (#10277)
  • 4acd720eb2a PIP-85 Add Schema Information to Message in Java Client API (#10476)
  • 56c2f6b3f95 cleaned some code in GenericJsonRecord (#10527)
  • b1ee56c5ce6 Add JsonRecordBuilder implementation (#10052)
  • 2932b1057a2 Add Schema.getNativeSchema (#10076)
  • 1264c476223 [Java client] Fix behaviour of Schema.AUTO_CONSUME() with KeyValueSchema and multi versions (#10492)
  • fe980e72bd4 Add JsonRecordBuilder implementation (#10052)
  • ea5b676372f [Security] Upgrade vertx to 3.9.7, addresses CVE-2018-12541 (#10261)
  • ff5924492d7 Java Client: MessageImpl - ensure that AutoConsumeSchema downloaded the schema before decoding the payload (#10248)
  • ec3ad85d1b8 Sink unwrap internal AutoConsumeSchema and allow to handle topics with KeyValue schema (#10211)
  • 72295b67bee GenericObject: handle KeyValue with SEPARATED encoding and add more tests (#10186)
  • b116efd0944 Pulsar IO: Allow to develop Sinks that support Schema but without setting it at build time (Sink) (#10034)
  • ab3caca169c GenericObject - support KeyValue in Message#getValue() (#10107)
  • 23b28593899 Implement GenericObject - Allow GenericRecord to wrap any Java Object (#10057)
  • 6fe93f50a8c Add presto password authenticators plugin to enable password auth in Pulsar SQL
  • 5bc23c6aead Enable Conscrypt for Jetty in the Broker and in the Proxy (#10541)
  • e275813efc7 Bump Debezium version to latest in series
  • 56e127d30d9 [Branch-2.7] Fix deadlock on Monitoring thread blocked by LeaderService.isLeader() (#10512)
  • 421fd828a02 [pulsar-functions] enhance kubernetes manifest customizer with default options (#9445)
  • 186e63b27bf Changing default function subscription to be function name, not FQFN
  • b56a6d86f9d Always return from trigger even if read from output topic times out
  • c9e071cca72 Update path for TTL config in Java 11
  • 830d79803e3 Switch Docker image to Java 11
  • d6f953893fa upgrade presto version to 334 and JDK11
  • 65a1acd7334 auth token for debezium and kafka connect adaptor