4.0.4

Addax 4.0.4 has released!

General

• Add how to debug guide dc3973d
• Add Excel Reader plugin (#346)
• Add Excel Writer plugin (#350)
• Bump logback.version from 1.2.5 to 1.2.6 (#349)
• Fix incorrect result when set value with SQLite date type (#354)
• Fix NullPoint exception occurred when reading a null column (#353)
• Bump junit-jupiter-api from 5.7.2 to 5.8.0 (#352)

Reader plugins

data reader

• Add datareader package to project's target folder (#321)

dbf reader

• Bump javadbf from 1.13.1 to 1.13.2 (#323)

rdbms reader

• Fix load JDBC driver failure (#334)

redis reader

• Bump jedis from 3.6.3 to 3.7.0 (#328)

sqlite reader

• Bump sqlite-jdbc from 3.36.0.2 to 3.36.0.3 (#324)

writer plugins

dbf writer

• Bump javadbf from 1.13.1 to 1.13.2 (#323)

greenplum writer

• Fix ClassCastException in CopyWorker#call (#325)
• Optimize write performance (#331)

rdbms writer

• Fix load JDBC driver failure (#334)

redis writer

• Bump jedis from 3.6.3 to 3.7.0 (#328)

sqlite writer

• Bump sqlite-jdbc from 3.36.0.2 to 3.36.0.3 (#324)

The sha256sum of the addax-4.0.4.tar.gz file is 67b94386726c96e85b0441fc592b78af6e7537f9764f76d4957b83cd8ca5ce5b

4.0.3

Addax 4.0.3 has released!

Download

1. You can download pre-compiled binary files from Addax-4.0.3 if you are in Chinese mainland, otherwise
2. Download from the following attachment addax-4.0.3.tar.gz

md5sum: 19766c2577b46bd5b22d63a502f5f5dd

Note: Because Aliyun drive does not support the sharing of compressed files currently, So, I add a .jpg suffix for each binary file, you need to remove this suffix before you can use the

Release notes

General

• Add datareader plugin, generate data to meet specified rule requirements for populating data in development and testing (#291)
• Introduce new sqlitereader plugin (#305)
• Introduce a new sqlitewriter plugin (#306)
• Optimize compress type process (#318)

Reader plugins

dbffile reader

• Rename to dbfreader 40d0d51

ftp reader

• Fix read failure when using gzip -c to compress multiple files (#316 )

kudu reader

• Allow specify columns instead of all columns (#297 #295 #252)
• Add support for configuring where to set more filters (#298)

sqlite reader

• Bump sqlite-jdbc from 3.36.0.1 to 3.36.0.2 (#311)

stream reader

• Allow specify scale when configure a random double (#288)

txtfile reader

• Fix read failure when using gzip -c to compress multiple files (#316 )

Writer plugins

dbffile writer

• Rename to dbfwriter c6adb3f

kudu writer

• Fixed data lost without flushing cache, thanks @niexiongchao (#294)

sqlite writer

• Bump sqlite-jdbc from 3.36.0.1 to 3.36.0.2 (#311)

4.0.2

4.0.2 has released!

download

1. You can download pre-compiled binary files from Addax-4.0.2 if you are in Chinese mainland, otherwise
2. Download from addax-4.0.2.tar.gz

md5sum: cd3a3d6d0c79cbd3bcd259ebb47acbc5

Note: Because Aliyun drive does not support the sharing of compressed files currently, So, I add a .jpg suffix for each binary file, you need to remove this suffix before you can use them

Release notes

General

• Add shrink package size script (#262)
• Add support for docker #265
• Update doc: Add JDBC-RESTful driver guide for TDengine database e847736
• Add support for writing a zip-compressed file (#282)

Reader plugins

hbase11xsql reader

• Change dependencies for reduce the size by half a1d548c

tdengine reader

• Bump taos-jdbcdriver from 2.0.32 to 2.0.33 (#261)
• Fix null pointer exception when not setting fetchSize item (#270)

Writer plugins

ftp writer

• Add support for zip compression format (#282)
• Add support for user authentication with public key (#283)

tdengine writer

• Bump taos-jdbcdriver from 2.0.32 to 2.0.33 (#261)
• Fix incorrect columnIndex (#272 , #277)

txtfile writer

• Add support for zip compression format (#282)

It has 3 attachements

• Source code (zip)
• Source code (tar.gz)
• addax-4.0.2.tar.gz (pre-compiled binary package)

4.0.1

4.0.1 has released!

The updated version 4.0.1 has relased

download

You can download pre-compiled binary files from Addax-4.0.1

Note: Because Aliyun drive does not support the sharing of compressed files currently, So, I add a .jpg suffix for each binary file, you need to remove this suffix before you can use them

Release notes

General

• The program run script is changed to addax.sh to remove the dependency on Python (#253)
• Strip part of the methods of the StorageReaderUtil class as a separate class, which has reduced the dependency of other plugins on plugin-storage-util (#227)
• addax.py remove unused options, remove --mode, --jobid, --reader, --writer
• Bump commons-compress from 1.20 to 1.21 (#232)
• Change project structure (#234)
• Bump commons-io from 2.10.0 to 2.11.0 (#236)
• Extract the configuration items and default values used by each plugin into a common class to reduce redundant code (#239)
• Bump logback.version from 1.2.3 to 1.2.4 (#243)
• Bump slf4j-api from 1.7.31 to 1.7.32 (#246)
• Fixed some typos (#250)

Reader plugins

dbf file reader

• Fixed missing 3rd dependency jars ( #225)
• Fixed null pointer exception when reading null value ( #229 )
• Add support for wildcard to column item (#235)

hbase11xsql reader

• Fix the problem that connecting hbase fails when kerberos is enabled (#241)

mongodb reader

• Bump mongo-java-driver from 3.12.8 to 3.12.9 (#238)

redis reader

• Bump redis-replicator from 3.5.4 to 3.5.5 (#221)
• Bump jedis from 3.6.2 to 3.6.3 (#247)

stream reader

• Add support for self-increment of time data type (#237)

textfile reader

• Add support for get column by name to txtfilereader plugin (#251, #254)

Writer plugins

hbase11x writer

• Add support for login with kerberos (#245)

hbase11xsql writer

• Improve the performance of writing (#242)

hbase20xsql writer

• Add support for kerberos authentication and phoenix thick connection mode (#248)

mongodb writer

• Bump mongo-java-driver from 3.12.8 to 3.12.9 (#238)

redis writer

• Bump jedis from 3.6.2 to 3.6.3 (#247)

4.0.0

This is the first release under the new brand name

Highlights

1. HDFS reader plugin supports reading of binary data type
2. HDFS writer plugin supports writing of binary data type
3. Rewritten the implementation of ORC and PARQUET for more efficient execution
4. Rewrite the string type field splitting method using the database's own functions
5. Add support for auto guess splittable column (introduce new configure item autoPk)
6. Bump a dozen of 3rd dependencies to the updated version

该版本主要功能点

1. HDFS reader 插件支持读取 binary 数据类型
2. HDFS writer 插件支持写入 binary 数据类型，该数据可以从关系型数据库的 Blob 字段获取(Oracle 暂未支持）
3. 重写了 ORC 和 PARQUET 文件的读写代码，使用两个文件系统的直接API，提升读写效率
4. 重写了字符类型字段的分割函数，利用数据自身的功能提供了更合理的分割范围，而不是之前的随机字符串分割，并支持中文
5. 增加了读取关系型数据库表时自动猜测分割字段的功能（默认不启用），详见文档 说明
6. 大量第三方依赖包升级到最新版本

Details

c52715e Changes main program to addax.py
288fbdf Add tdengine reader/writer plugins to package #224
6c9f4b4 Bugfixed: exception occurred when move hdfs file from temporary location to destination
e582f76 Use phoenix-core instead of phoenix-client to reduce size
76d3062 Bugfix: string format error
86457ab Add correspond compress suffix
9c6e88f Bump postgresql from 42.2.22 to 42.2.23 (#218)
df8e8be Add correspond compress suffix
e4a04a8 Bump maven-javadoc-plugin from 3.2.0 to 3.3.0 (#211)
83221c3 Bump commons-net from 3.7.2 to 3.8.0 (#212)
7cbf4c6 Bump parquet-avro from 1.11.1 to 1.12.0 (#213)
6264fac Bump httpcore from 4.4.13 to 4.4.14 (#214)
40e6194 Rewrite the string type field splitting method using the database's own functions
18a6c3f Bump kudu-client from 1.8.0 to 1.15.0 (#190)
d8f3295 Bump nexus-staging-maven-plugin from 1.6.7 to 1.6.8 (#210)
91de3c2 Bump commons-lang3 from 3.11 to 3.12.0 (#209)
0e80d62 Bump commons-io from 2.8.0 to 2.10.0 (#208)
68f7e03 Bump snappy-java from 1.1.8 to 1.1.8.4 (#207)
4a638dc Bump taos-jdbcdriver from 2.0.18 to 2.0.32 (#204)
5c25819 Bump fastjson from 1.2.75 to 1.2.76 (#206)
58a8a57 Bump ognl from 3.2.18 to 3.2.21 (#203)
8fc41ad Bump joda-time from 2.9.7 to 2.10.10 (#205)
02b000b Bump maven-dependency-plugin from 3.1.2 to 3.2.0 (#199)
6802862 Bump jedis from 3.4.0 to 3.6.1 (#201)
62ea143 Bump error_prone_core from 2.4.0 to 2.7.1 (#202)
8fee29b Bump maven-gpg-plugin from 1.5 to 3.0.1 (#194)
49342f7 Bump postgresql from 42.2.19 to 42.2.22 (#195)
eb09cad Bump aircompressor from 0.17 to 0.19 (#196)
96c3c85 Bump druid from 1.2.4 to 1.2.6 (#197)
96e7e75 Add support for auto guess splittable column (#198)
920f93c Bump cassandra.jdbc.version from 3.10.2 to 3.11.0 (#192)
44e2d72 Bump avro from 1.10.1 to 1.10.2 (#188)
26d2783 Bump slf4j-api from 1.7.28 to 1.7.31 (#189)
f8d481b Bump json-path from 2.5.0 to 2.6.0 (#191)
b254c18 Bump javadbf from 1.11.2 to 1.13.1 (#186)
238fbe2 Bump fluent-hc from 4.5.12 to 4.5.13 (#187)
9516941 Bump httpcore-nio from 4.4.13 to 4.4.14 (#181)
ca07dca Bump redis-replicator from 3.4.2 to 3.5.4 (#148)
89feeb7 Bump mongo-java-driver from 3.12.7 to 3.12.8 (#142)
0320d50 Add test for decimal type
98286bb Allow all fields accept null value
52ace57 plugin hdfswriter: rewrite orc file schema generate method
ce8cfbf plugin hdfsreader: simplify parquet file reader codes
bd4aa41 Uses consistent variable definitions
0c14f86 plugin: dbffilewriter Add check for column configuration
8401fbc bugfix FileNotFoundException
73e8e03 Support for read/write Parquet file with binary type
1c0b80d Add some test cases
cbe64da plugin hdfsreader: Add support for reading binary data
4e008b9 plugin hdfswriter: Add support for writing binary data with OrcFile

3.2.5

• 24bdf31 Remove duplicate string
• 7b11021 Httpreader support sock proxy (#177)
• 6507fb9 bugfix: txtreader process blank or null value throws null point exception
• 9961d5e bugfix: Incorrect conversion of mysql year type
• 548adf7 bugfix: convert json to string will filter null value #171 (#172)
• 74a5ca1 Use connection pool for improve connection stable

Known issues

• #222

3.2.4

6c82981 Migrates common-lang to common-lang3
b68f42c Use connection pool for improve connection stable
4b61893 Remove un-necessary username and password
c5b308d Extracts IP/Host from all kinds of jdbc url
151314a Merge branch 'master' of github.com:wgzhao/DataX
eb2ddc6 Add support for reading compress files in jsonfilereader plugin
96c2f9b Add detail info when hive type cast failure
c8ebaeb Merge remote-tracking branch 'github/master'
24082b3 Fixed read only the first line via each file
ce07a1b improve retention-policy implement in InfluxDBWriter plugin #162
125f9d1 influxdb: fix column maybe not have time (#160)
9bebd7a Add retentionPolicy option in plugin InfluxDBWriter #162
810965e Fixed #157 javadoc syntax error
a04ccc3 Merge branch 'master' of github.com:wgzhao/DataX
f89ee15 Fixed #158 main class not found in Windows and in-correct encoding in Windows
0e3f4e8 Fixed protobuf in-compliant version
8e7f910 Fixed printf error
e10a0f0 Fixed javadoc syntax error #157
339ae8f 1. Fixed #156 exception occurred when writing Parquet file with null value 2. Optimize Decimal type
55079ba Fixed #155 jdbc java runtime version compliant exception

3.2.3

General Changes

• Bump protobuf-java from 2.5.0 to 3.15.5
• Bump mssql-jdbc from 8.4.0.jres8 to 9.2.1.jre15
• Add missing ElasticSearchReader package
• Add support for get version in datax.py

RDBMS Reader

• Fixed get XML data as string instead of SQLXML handler

RDBMS Writer

• Add support XML data type

Stream Reader

• Add field value incrment function

InfluxDB Reader

• Fixed job template file
• Add timeout configure items
• Use native influxDB API instead of influxdb-java (#153)

InfluxDB Writer

• Add timeout configure items

Textfile Reader

• Add support for more compress format (#154)

3.2.2

1. Fixed: unknown host exception when writing parquet file #135
2. Documentation streamreader and streamwriter
3. Add support for DorisDB writer #131
4. Add support more than one channel setup for InfluxDB #130
5. Add support for InfluxDB TAG data type #125
6. Add support for TDengine reader/writer #119
7. Add flattend index type

3.2.1

[maven-release-plugin] copy for tag 3.2.1