1.13.0

Agent

ISSUE	Summary
INLONG-10094	[Improve][Agent] The task record for data supplementation has not expired
INLONG-10107	[Improve][Agent] There is a bug in updating the module configuration
INLONG-10113	[Improve][Agent] Delete useless code
INLONG-10115	[Improve][Agent] Offset needs to be changed to save in string format because some data sources have non integer positions
INLONG-10187	[Improve][Agent] Need to handle situations where timeoffset is empty, default to no offset
INLONG-10189	[Improve][Agent] There is a risk of agent getting stuck after SDK initialization exception
INLONG-10191	[Improve][Agent] Delete useless code
INLONG-10210	[Improve][Agent] The installer needs to add a script for environment initialization
INLONG-10268	[Improve][Agent] Suggest not using task ID as the data version
INLONG-10281	[Improve][Agent] Real time collection of files with audit data errors
INLONG-10298	[Improve][Agent] The command related code is no longer useful, it is recommended to delete it
INLONG-10302	[Improve][Agent] The Task base class needs to add an interface that limits the number of instances obtained
INLONG-10318	[Feature][Agent] Add PostgreSQL data source for Agent
INLONG-10319	[Improve][Agent] Agent should get audit id from audit sdk
INLONG-10384	[Improve][Agent] Add functions to the Store interface to extend new storage plugins
INLONG-10399	[Improve][Agent] Add global configurations updater
INLONG-10410	[Improve][Agent] Add ZK plugin to save offset info
INLONG-10443	[Improve][Agent] Put Rocksdb into the plugins module
INLONG-10446	[Improve][Agent] Adjusting audit SDK address settings
INLONG-10476	[Improve][Agent] The audit address should only be obtained from the manager
INLONG-10535	[Improve][Agent] Support minute level tasks
INLONG-10542	[Improve][Agent] Remove the deleted watch directions
INLONG-10547	[Improve][Agent] Provide a clear prompt after the mq type is incorrect
INLONG-10564	[Improve][Agent] Request configuration with md5 included
INLONG-10598	[Improve][Agent] Delete excess unit tests
INLONG-10611	[Improve][Agent] Update configuration only when the version number is increased
INLONG-10633	[Improve][Agent] The initialization function of AuditUtils needs to pass in the configuration
INLONG-10645	[Improve][Agent] Installer needs to add process protection
INLONG-10650	[Improve][Agent] When the installer updates the configuration, it is necessary to first determine the version

Dashboard

ISSUE	Summary
INLONG-10179	[Bug][Dashboard] There are redundant All types in cluster management
INLONG-10226	[Bug][Dashboard] Audit items cannot be searched
INLONG-10256	[Improve][DashBoard] Modify the data source IP item of the file data source form in the data access module
INLONG-10314	[Improve][DashBoard] Add an operation time to the operation log table
INLONG-10347	[Improve][Dashboard] Add Agent IP field to PostgreSQL data source
INLONG-10350	[Bug][Dashboard] Audit item search failed
INLONG-10377	[Improve][DashBoard] add Source Data Field Template
INLONG-10394	[Improve][DashBoard] Agent page modifies the version and sets the default values for the commands
INLONG-10409	[Feature][Dashboard] Support installing agents by SSH key-based authentication
INLONG-10417	[Bug][Dashboard] There is an issue with the MODE display on the approval screen, and data synchronization is not displayed
INLONG-10419	[Feature][Dashboard] Automatically switch tenants when opening a page with groupId
INLONG-10449	[Bug][Dashboard] The field template for selecting a new data flow does not take effect
INLONG-10468	[Feature][Dashboard] Audit data showing totals and variances
INLONG-10500	[Improve][Dashboard] When you edit tenants in a template, you need to change the scope to hide them
INLONG-10501	[Improve][Dashboard] Modify component type to be more intuitive
INLONG-10504	[Improve][Dashboard] Added details button to tag management
INLONG-10584	[Improve][Dashboard] New cluster type adds sortkafka types
INLONG-10592	[Improve][Dashboard] When there are too many selections in the drop-down box, omit some of them
INLONG-10614	[Improve][Dashboard] The template list does not need to display Id
INLONG-10617	[Improve][Dashboard] The mq type should not exist on the data synchronization page
INLONG-10620	[Bug][Dashboard] Page error occurs wrong
INLONG-10640	[Improve][Dashboard] Approval page display item modification
INLONG-10651	[Improve][Dashboard] File Type data stream supports minute-level periods
INLONG-10681	[Bug][Dashboard] There is a nesting problem in moduleIdList
INLONG-10691	[Improve][Dashboard] Added monitoring and auditing page

Tube

ISSUE	Summary
INLONG-10126	[Bug][tubemq-server] Maven sleepycat je.version 7.3.7 can not found in maven repository

DataProxy

ISSUE	Summary
INLONG-10066	[Improve][DataProxy] Optimize the related configuration settings in the CommonConfigHolder.java file
INLONG-10067	[Improve][DataProxy] Simplify the configuration and acquisition of the Manager address
INLONG-10080	[Improve][DataProxy] DataProxy implementation optimization
INLONG-10081	[Improve][DataProxy] Modify the data format of metadata saved in the metadata.json file
INLONG-10102	[Improve][DataProxy] Adjust the position where Source calls addSourceReportInfo()
INLONG-10111	[Improve][DataProxy] Add auditVersion field processing
INLONG-10313	[Improve][DataProxy] Replace audit ID macro with audit API
INLONG-10441	[Improve][DataProxy] DataProxy supports obtaining Audit-Proxy through InLong Manager

Manager

ISSUE	Summary
INLONG-10056	[Feature][Manager] Support new manager plugin for flink 1.18
INLONG-10071	[Improve][Manager] Supplementary statement trailing semicolon in apache_inlong_manager.sql
INLONG-10074	[Bug][Manager] Error in querying audit information based on IP address
INLONG-10076	[Bug][Manager] Data type 'doris' not support
INLONG-10096	[Improve][Manager] Support installing agents by SSH
INLONG-10105	[Bug][Manager]Fix sort standalone get kafka config error
INLONG-10135	[Improve][Manager] Move inlongCompressType to clustertag configuration
INLONG-10136	[Improve][Manager] Clean up ElasticSearch and ClickHouse audit source query function
INLONG-10141	[Improve][Manager] Data preview supports returning header and specific field information for inlong msg v1
INLONG-10150	[Improve][Manager] Remove metrics. audit. proxy. hosts from Flink sort plugins. properties
INLONG-10169	[Improve][Manager]Support configure sort cluster for kafka
INLONG-10176	[Bug][Manager] Table creation statement error
INLONG-10181	[Improve][Manager] Remove useless exceptions for DataNodeOperator.getNodeConfig
INLONG-10197	[Improve][Manager] Support OpenAPI for querying audit data
INLONG-10200	[Improve][Manager] Define module type mapping relationships
INLONG-10204	[Feature][Manager] Kafka sink supports automatic allocation of sort standalone cluster
INLONG-10233	[Improve][Manager] Report heartbeat removal port restriction
INLONG-10245	[Improve][Manager] Support setting audit version for file collection
INLONG-10247	[Feature][Manager] Support schedule information management for offline sync
INLONG-10249	[Bug][Manager]Duplicate data appears during data preview
INLONG-10252	[Bug][Manager] The audit ip which taskmanager received was wrong
INLONG-10260	[Bug][Manager] Correct the wrong create table statement
INLONG-10261	[Feature][Manager] Support installing agents by SSH key-based authentication
INLONG-10265	[Bug][Manager] Correct wrong starrocks create table statement
INLONG-10266	[Bug][Manager] Password is overwritten when adding field information
INLONG-10270	[Improve][Manager] Data source tasks allow for multiple IPs
INLONG-10277	[Improve][Manager] Support calling API to refresh cluster config
INLONG-10279	[Improve][Manager] Support compress unified configuration
INLONG-10283	[Bug][Manager] Migration of groups does not support multiple groups migrating to the same tenant
INLONG-10284	[Improve][Manager] Change the default flink version from 1.13 to 1.15
INLONG-10290	[Improve][Manager] Prohibit groups that have not been successfully configured from obtaining dataproxy addresses
INLONG-10300	[Improve][Manager] Allow unsubmitted groups to modify mq type
INLONG-10305	[Improve][Manager] Delete k8s related parameters in file collection
INLONG-10324	[Bug][Manager] UseExtendedFields in Stream incorrectly overwritten
INLONG-10326	[Improve][Manager] Support obtaining tenant information based on groupId
INLONG-10328	[Improve][Manager] Support automatic synchronization of stream fields to sink
INLONG-10330	[Improve][Manager] Support field template management
INLONG-10335	[Improve][Manager] Use audit sdk to obtain audit ID
INLONG-10345	[Bug][Manager] Error in obtaining audit id value
INLONG-10351	[Bug][Manager] No auditname returned when obtaining audit information
INLONG-10353	[Improve][Manager] refactor code for building and submitting flink job
INLONG-10360	[Improve][Manager] Combine schedule state transition with group operations
INLONG-10362	[Improve][Manager] Simplify code for InLong group management
INLONG-10363	[Improve][Manager] Support template multi tenant management
INLONG-10368	[Improve][Manager] Data preview supports obtaining message attribute information
INLONG-10370	[Improve][Manager] Support configuration of kV data format
INLONG-10373	[Improve][Manager] Manager client support template operation
INLONG-10375	[Improve][Manager] Add field mapping strategy for CLS, StarRocks and Elasticsearch
INLONG-10382	[Improve][Manager] Support obtaining node information through unique keys
INLONG-10388	[Improve][Manager] Provide an interface to obtain the audit proxy address
INLONG-10391	[Improve][Manager] Supports configuring zk clusters and issuing zk addresses to agents
INLONG-10395	[Feature][Manager] Add interface for schedule client and engine
INLONG-10396	[Feature][Manager] Support build-in schedule base on quartz
INLONG-10405	[Bug][Manager] Iceberg field type mapping error
INLONG-10407	[Improve][Manager] Increase the interval for determining heartbeat timeout
INLONG-10413	[Improve][Manager] Support for configuring built-in fields for mysql and kafka
INLONG-10415	[Improve][Manager] Kafka source supports configuring wraptype
INLONG-10423	[Improve][Manager] Modify unified configuration related classes and interfaces
INLONG-10425	[Bug][Manager] The kafka sink defines duplicate attributes
INLONG-10430	[Bug][Manager] Check update error of unified configuration
INLONG-10432	[Improve][Manager] Delete unused method getMetaConfig
INLONG-10434	[Improve][Manager] Allow creating cluster nodes without setting port
INLONG-10436	[Improve][Manager] move scheudle configuration from stream to group level
INLONG-10438	[Improve][Manager] GetConfig does not throw an exception when obtaining the zk address fails
INLONG-10452	[Improve][Manager] Delete the method of querying audit information through MySQL
INLONG-10455	[Bug][Manager] Set KafkaDataNode ack fail
INLONG-10459	[Feature][Manager] Support schedule instance callback to submit Flink batch job
INLONG-10472	[Improve][Manager] Use audit SDK to obtain audit proxy URL
INLONG-10474	[Feature][Manager] Restrict sortTaskName and sortConsumeName the same with datanodeName when migrate Inlong Group
INLONG-10484	[Improve][Manager] Refactor code in manager-service module
INLONG-10487	[Bug][Manager] Not return success ListenerResult
INLONG-10495	[Bug][Manager] ScanStartupSubStartOffset is set to null in pulsar extranode
INLONG-10498	[Improve][Manager] Template supports modifying associated tenants
INLONG-10505	[Bug][Manager] Data preview cannot recognize null values in the data
INLONG-10506	[Improve][Manager] When obtaining a template, return information such as creator and modifier
INLONG-10512	[Improve][Manager] Support preview of data in kv data type
INLONG-10514	[Feature][Manager] Support built-in schedule engine trigger submitting of Flink batch job
INLONG-10516	[Improve][Manager] Add jdbc connectors on flink 1.15 module to manager image
INLONG-10520	[Bug][Manager] Protocol Type reported by dataproxy not correctly accepted
INLONG-10543	[Improve][Manager] The delimiter and other configurations in CLS and ES sink are obtained from the stream
INLONG-10558	[Improve][Manager] Support determining whether to issue agent tasks based on the MD5 value
INLONG-10561	[Feature][Manager] Support configrations for bounded source
INLONG-10562	[Feature][Manager] SortConfig supports set start and stop consume time
INLONG-10589	[Improve][Manager] Rename OfflineJobSubmitRequest and remove log in controller
INLONG-10601	[Improve][Manager] Optimize the agent task configuration process
INLONG-10625	[Bug][Manager] The responsible person cannot modify the template configuration
INLONG-10635	[Improve][Manager] Optimize the installer configuration process
INLONG-10638	[Improve][Manager] Data preview supports filtering function
INLONG-10648	[Bug][Manager] The get sort config interface returns an error status code
INLONG-10689	[Improve][Manager] Support querying metric information

SDK

ISSUE	Summary
INLONG-10109	[Feature][SDK] Support to transform from Json protocol to CSV/KV protocol by single SQL
INLONG-10117	[Feature][SDK] Support to transform from PB protocol to CSV/KV protocol by single SQL
INLONG-10129	[Feature][SDK] Transform SQL support +-*/ operations
INLONG-10130	[Feature][SDK] Transform SQL support string concat function
INLONG-10154	[Feature][SDK] Support to transform CSV/KV data to CSV/KV data without field list configuration
INLONG-10213	[Feature][SDK] SortSDK support unified sort configuration
INLONG-10221	[Improve][SDK] DataProxy SDK of cpp supports automatic installation of log4cplus components
INLONG-10291	[Bug][SDK] Incorrect initializing of gnet in Golang SDK
INLONG-10292	[Bug][SDK] Panic in connpool.UpdateEndpoints() of Golang SDK
INLONG-10427	[Feature][SDK] The Go SDK supports authentication for Manager access
INLONG-10457	[Bug][SDK] Fix auth spelling errors
INLONG-10522	[Feature][SDK] SortSDK support assgin subscription
INLONG-10531	[Feature][SDK] Add InLong Dataproxy Python SDK based on C++ SDK
INLONG-10532	[Improve][SDK] Add InLong Dataproxy Python SDK sample
INLONG-10534	[Umbrella][SDK] InLong Dataproxy Python SDK
INLONG-10603	[Feature][SDK] Transform SQL support arithmetic functions(Including power, abs, sqrt and ln)
INLONG-10607	[Feature][SDK] Transform SQL support arithmetic functions(Including log10, log2, log and exp)
INLONG-10652	[Improve][SDK] Inlong Transform support for generics
INLONG-10668	[Improve][SDK] Add rebalance and recover support of conn pool in Golang SDK
INLONG-10670	[Bug][SDK] Potential block in Golang SDK
INLONG-10676	[Improve][SDK] Fix type asserttion and type switch warnings in Golang SDK
INLONG-10677	[Improve][SDK] Bump up modules in Golang SDK
INLONG-10675	[Improve][SDK] Use exponential backoff instead of linear backoff retrying in Golang SDK

Sort

ISSUE	Summary
INLONG-8464	[Feature] [Sort] Add JDBC connector on Flink 1.15
INLONG-10053	[Feature][Sort] Support flink-connector-pulsar based on flink 1.18
INLONG-10054	[Feature][Sort] Add dependencies for flink 1.18
INLONG-10055	[Feature][Sort] Support flink-connector-jdbc based on flink 1.18
INLONG-10069	[Feature][Sort] Support audit metrics for sort-connector-pulsar-1.18
INLONG-10091	[Feature][Sort] Add json format for 1.18
INLONG-10144	[Feature][Sort] Redis connectors support audit ID
INLONG-10152	[Improve][Sort] Refactor MetricOption code structure.
INLONG-10159	[Feature][Sort] Kafka connector support audit ID
INLONG-10164	[Umbrella][Sort] SortStandalone support unified SortConfig
INLONG-10173	[Feature][Sort] SortStandalone support request unified configuration
INLONG-10183	[Feature][Sort] MongoDB connector support audit ID
INLONG-10193	[Feature][Sort] Postgres connector support audit ID
INLONG-10194	[Feature][Sort] Sqlserver connector support audit ID
INLONG-10208	[Feature][Sort] ClsSink support unified configuration
INLONG-10224	[Feature][Sort] Unified configuration check utils
INLONG-10228	[Feature][Sort] PulsarSink support unified configuration
INLONG-10229	[Feature][Sort] EsSink support unified configuration
INLONG-10230	[Feature][Sort] KafkaSink support unified configuration
INLONG-10257	[Improve][Sort] Upgrade flink version from 1.13.6 to 1.15.4
INLONG-10272	[Improve][Sort] Unified configuration check utils support check latest config
INLONG-10296	[Bug][Sort] Connectors AuditOperator was not serialized
INLONG-10297	[Bug][Sort] The audit operator in mysql connector cannot be serialized and the job cannot submit to jobmanager
INLONG-10311	[Feature][Sort] TubeMQ source support report audit information exactly once
INLONG-10312	[Feature][Sort] Iceberg sink support report audit information exactly once
INLONG-10317	[Feature][Sort] Kafka Source support report audit information exactly once
INLONG-10323	[Feature][Sort] Support Kv deserialization in sort module
INLONG-10338	[Bug][Sort] Sqlserver connector's AuditOperator was not serialized
INLONG-10339	[Bug][Sort] PostgreSQL connector's AuditOperator was not serialized
INLONG-10340	[Bug][Sort] MongoDB connector's AuditOperator was not serialized
INLONG-10355	[Feature][Sort] Iceberg source support report audit information exactly once
INLONG-10357	[Feature][Sort] Starrocks sink support report audit information exactly once
INLONG-10358	[Feature][Sort] Pulsar source support report audit information exactly once
INLONG-10401	[Improve][Sort] Add metadata for Mysql connector and relocate debezium dependencies
INLONG-10489	[Bug][Sort] Mongodb2StarRocksTest sometime occur error when in workflow
INLONG-10492	[Bug][Sort] Init failure of pulsar connector
INLONG-10508	[Bug][Sort] Fix pulsar connector flink 1.15 parameter cannot keep consistent with flink 1.13
INLONG-10526	[Improve][Sort] ClsSink support switch metadata acquire mode
INLONG-10527	[Improve][Sort] EsSink support switch metadata acquire mode
INLONG-10528	[Improve][Sort] KafkaSink support switch metadata acquire mode
INLONG-10529	[Improve][Sort] PulsarSink support switch metadata acquire mode
INLONG-10530	[Umbrella][Sort] Sortstandalone support switch metadata acquire mode
INLONG-10540	[Feature][Sort] SortStandalone unified configuration metric reporter
INLONG-10555	[Feature][Sort] SortStandalone support report the difference between two configuration
INLONG-10560	[Feature][Sort] Support bounded pulsar source
INLONG-10568	[Improve][Sort] The starrocks connector UNKNOWN datatype handle method need to change
INLONG-10573	[Improve][Sort] Pulsar source connector report audit attach input time
INLONG-10575	[Improve][Sort] Mysql source support report audit information exactly once
INLONG-10577	[Feature][Sort] Simplified SortStandalone SortSdkSource
INLONG-10594	[Improve][Sort] Provide default kafka producer configuration
INLONG-10597	[Improve][Sort] Provide default pulsar producer configuration
INLONG-10604	[Bug][Sort] NPE when unified configuration is not exits
INLONG-10609	[Improve][Sort] PostgreSQL source support report audit information exactly once
INLONG-10610	[Improve][Sort] MongoDB source support report audit information exactly once
INLONG-10623	[Feature][Sort] The Pulsar connector on flink1.18 not set audit time, it should be set as input time through 'consume_time' metadata field
INLONG-10628	[Feature][Sort] The end2end test env on flink1.18 has not implemented
INLONG-10630	[Feature][Sort] SQL Server source support report audit information exactly once
INLONG-10655	[Improve][Sort] Kafka and Pulsar Sink support parse stream separator
INLONG-10682	[Improve][Sort] Pulsar source sending audit information support exactly once and helper.validateExcept should add ExtractNode.INLONG_MSG
INLONG-10696	[Improve][Sort] StarRocks sink supports ignore json parse error
INLONG-10694	[Bug][Sort] Mysql-cdc source fails to convert timestamp type

Audit

ISSUE	Summary
INLONG-10085	[Improve][Audit] Optimize the performance of audit-service
INLONG-10089	[Improve][Audit] Adjust the service configuration of Audit
INLONG-10122	[Improve][Audit] Update the apache_inlong_audit.sql to apache_inlong_audit_mysql.sql
INLONG-10132	[Improve][Audit] Clean up Elasticsearch and ClickHouse related code of audit-store
INLONG-10147	[Improve][Audit] Less usage costs when the Audit uses the MySQL as storage
INLONG-10171	[Improve][Audit] Update the standalone deploy scripts for the audit service
INLONG-10201	[Improve][Audit] Rename configuration variables
INLONG-10225	[Improve][Audit] Audit SDK provides the ability to automatically assign and manage Audit ID
INLONG-10242	[Improve][Audit] Audit SDK compatible with InLong Manager to manage audit items
INLONG-10263	[Improve][Audit] Solve the conflict between the jdbc.url parameter of Audit Store and the container environment variable
INLONG-10274	[Improve][Audit] The OpenAPI of Audit Service returns the average transmission time
INLONG-10306	[Improve][Audit] Compatible with scenarios where the Audit Tag is empty
INLONG-10321	[Improve][Audit] Audit supports the Audit Proxy service discovery and management
INLONG-10321	[Improve][Audit] Audit supports the Audit Proxy service discovery and management
INLONG-10365	[Improve][Audit] Optimizing TCP sticky packets may lead to duplication of audit data
INLONG-10379	[Improve][Audit] Add HDFS Audit items in the Audit SDK
INLONG-10387	[Improve][Audit] Audit SDK supports obtaining Audit-Proxy capabilities through InLong Manager
INLONG-10402	[Improve][Audit] Audit Service supports the hourly Audit data one day ago
INLONG-10461	[Bug][Audit] Cause HttpHostConnectException for manager
INLONG-10470	[Improve][Audit] Optimize Audit Proxy configuration update retry logic
INLONG-10478	[Improve][Audit] Default to enable audit for the Docker and Standalone deployment
INLONG-10480	[Improve][Audit] Audit Service automatically manages MySQL partitions
INLONG-10481	[Improve][Audit] Optimize Audit domain management
INLONG-10687	[Improve][Audit] Independent the Audit items of Agent from module reconciliation

Other

ISSUE	Summary
INLONG-10051	[Improve][Github] improve the pull request template
INLONG-10057	[Feature][Distribution] Support assembling flink 1.18 dependencies and connectors
INLONG-10064	[Feature][Tool] Support multi-version flinks in inlong-dev-toolkit
INLONG-10120	[Improve][CI] Update the actions/upload-artifact to V4
INLONG-10124	[Bug][deploy] In 1.12 the single-machine deployment fails
INLONG-10125	[Bug][Script] In 1.11 single-machine deployment, the agent module cannot be started
INLONG-10138	[Improve][CI] Node.js 16 actions are deprecated
INLONG-10160	[Improve][CVE] Elasticsearch vulnerable to Uncontrolled Resource Consumption
INLONG-10161	[Improve][CVE] Golang protojson.Unmarshal function infinite loop when unmarshaling certain forms of invalid JSON
INLONG-10162	[Improve][CVE] net/http, x/net/http2: close connections when receiving too many headers
INLONG-10205	[Improve][Script] Add the metrics.audit.proxy.hosts for standalone deployment
INLONG-10331	[Improve][CVE] Decompressors can crash the JVM and leak memory content in Aircompressor
INLONG-10582	[Improve][ASF] Disable merge and rebase merge