diff --git a/README.md b/README.md index b3a2a06..04322f7 100644 --- a/README.md +++ b/README.md @@ -19,11 +19,11 @@ For information on Siddhi and i ## Latest API Docs -Latest API Docs is 2.0.5. +Latest API Docs is 2.0.6. ## Features -* cdc *(Source)*

The CDC source receives events when change events (i.e., INSERT, UPDATE, DELETE) are triggered for a database table. Events are received in the 'key-value' format.

There are two modes you could perform CDC: Listening mode and Polling mode.

In polling mode, the datasource is periodically polled for capturing the changes. The polling period can be configured.
In polling mode, you can only capture INSERT and UPDATE changes.

On listening mode, the Source will keep listening to the Change Log of the database and notify in case a change has taken place. Here, you are immediately notified about the change, compared to polling mode.

The key values of the map of a CDC change event are as follows.

For 'listening' mode:
    For insert: Keys are specified as columns of the table.
    For delete: Keys are followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.
    For update: Keys are followed followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.

For 'polling' mode: Keys are specified as the columns of the table.#### Preparations required for working with Oracle Databases in listening mode

Using the extension in Windows, Mac OSX and AIX are pretty straight forward inorder to achieve the required behaviour please follow the steps given below

  - Download the compatible version of oracle instantclient for the database version from [here](https://www.oracle.com/database/technologies/instant-client/downloads.html) and extract
  - Extract and set the environment variable LD_LIBRARY_PATH to the location of instantclient which was exstracted as shown below
  

+* cdc *(Source)*
 
The CDC source receives events when change events (i.e., INSERT, UPDATE, DELETE) are triggered for a database table. Events are received in the 'key-value' format.

There are two modes you could perform CDC: Listening mode and Polling mode.

In polling mode, the datasource is periodically polled for capturing the changes. The polling period can be configured.
In polling mode, you can only capture INSERT and UPDATE changes.

On listening mode, the Source will keep listening to the Change Log of the database and notify in case a change has taken place. Here, you are immediately notified about the change, compared to polling mode.

The key values of the map of a CDC change event are as follows.

For 'listening' mode: 
    For insert: Keys are specified as columns of the table.
    For delete: Keys are followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.
    For update: Keys are followed followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.

For 'polling' mode: Keys are specified as the columns of the table.#### Preparations required for working with Oracle Databases in listening mode

Using the extension in Windows, Mac OSX and AIX are pretty straight forward inorder to achieve the required behaviour please follow the steps given below

  - Download the compatible version of oracle instantclient for the database version from [here](https://www.oracle.com/database/technologies/instant-client/downloads.html) and extract
  - Extract and set the environment variable LD_LIBRARY_PATH to the location of instantclient which was exstracted as shown below
  
     export LD_LIBRARY_PATH=<path to the instant client location>
   

  - Inside the instantclient folder which was download there are two jars xstreams.jar and ojdbc<version>.jar convert them to OSGi bundles using the tools which were provided in the <distribution>/bin for converting the ojdbc.jar use the tool spi-provider.sh|bat and for the conversion of xstreams.jar use the jni-provider.sh as shown below(Note: this way of converting Xstreams jar is applicable only for Linux environments for other OSs this step is not required and converting it through the jartobundle.sh tool is enough)
  
     ./jni-provider.sh <input-jar> <destination> <comma seperated native library names>
diff --git a/docs/api/2.0.6.md b/docs/api/2.0.6.md
new file mode 100644
index 0000000..d67ded9
--- /dev/null
+++ b/docs/api/2.0.6.md
@@ -0,0 +1,248 @@
+# API Docs - v2.0.6
+
+!!! Info "Tested Siddhi Core version: *5.1.7*"
+    It could also support other Siddhi Core minor versions.
+
+## Source
+
+### cdc *(Source)*
+

+
The CDC source receives events when change events (i.e., INSERT, UPDATE, DELETE) are triggered for a database table. Events are received in the 'key-value' format.

There are two modes you could perform CDC: Listening mode and Polling mode.

In polling mode, the datasource is periodically polled for capturing the changes. The polling period can be configured.
In polling mode, you can only capture INSERT and UPDATE changes.

On listening mode, the Source will keep listening to the Change Log of the database and notify in case a change has taken place. Here, you are immediately notified about the change, compared to polling mode.

The key values of the map of a CDC change event are as follows.

For 'listening' mode: 
    For insert: Keys are specified as columns of the table.
    For delete: Keys are followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.
    For update: Keys are followed followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.

For 'polling' mode: Keys are specified as the columns of the table.#### Preparations required for working with Oracle Databases in listening mode

Using the extension in Windows, Mac OSX and AIX are pretty straight forward inorder to achieve the required behaviour please follow the steps given below

  - Download the compatible version of oracle instantclient for the database version from [here](https://www.oracle.com/database/technologies/instant-client/downloads.html) and extract
  - Extract and set the environment variable LD_LIBRARY_PATH to the location of instantclient which was exstracted as shown below
  
+    export LD_LIBRARY_PATH=<path to the instant client location>
+  

  - Inside the instantclient folder which was download there are two jars xstreams.jar and ojdbc<version>.jar convert them to OSGi bundles using the tools which were provided in the <distribution>/bin for converting the ojdbc.jar use the tool spi-provider.sh|bat and for the conversion of xstreams.jar use the jni-provider.sh as shown below(Note: this way of converting Xstreams jar is applicable only for Linux environments for other OSs this step is not required and converting it through the jartobundle.sh tool is enough)
  
+    ./jni-provider.sh <input-jar> <destination> <comma seperated native library names>
+  

  once ojdbc and xstreams jars are converted to OSGi copy the generated jars to the <distribution>/lib. Currently siddhi-io-cdc only supports the oracle database distributions 12 and above

See parameter: mode for supported databases and change events.

+

+Syntax
+
+```
+@source(type="cdc", url="", mode="", jdbc.driver.name="", username="", password="", pool.properties="", datasource.name="", table.name="", polling.column="", polling.interval="", operation="", connector.properties="", database.server.id="", database.server.name="", wait.on.missed.record="", missed.record.waiting.timeout="", @map(...)))
+```
+
+QUERY PARAMETERS
+
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+    
+        
+        
+        
+        
+        
+        
+    
+
NameDescriptionDefault ValuePossible Data TypesOptionalDynamic
url
The connection URL to the database.
F=The format used is: 'jdbc:mysql://<host>:<port>/<database_name>' 
STRINGNoNo
mode
Mode to capture the change data. The type of events that can be received, and the required parameters differ based on the mode. The mode can be one of the following:
'polling': This mode uses a column named 'polling.column' to monitor the given table. It captures change events of the 'RDBMS', 'INSERT, and 'UPDATE' types.
'listening': This mode uses logs to monitor the given table. It currently supports change events only of the 'MySQL', 'INSERT', 'UPDATE', and 'DELETE' types.
listeningSTRINGYesNo
jdbc.driver.name
The driver class name for connecting the database. **It is required to specify a value for this parameter when the mode is 'polling'.**
STRINGYesNo
username
The username to be used for accessing the database. This user needs to have the 'SELECT', 'RELOAD', 'SHOW DATABASES', 'REPLICATION SLAVE', and 'REPLICATION CLIENT'privileges for the change data capturing table (specified via the 'table.name' parameter).
To operate in the polling mode, the user needs 'SELECT' privileges.
STRINGNoNo
password
The password of the username you specified for accessing the database.
STRINGNoNo
pool.properties
The pool parameters for the database connection can be specified as key-value pairs.
STRINGYesNo
datasource.name
Name of the wso2 datasource to connect to the database. When datasource name is provided, the URL, username and password are not needed. A datasource based connection is given more priority over the URL based connection.
 This parameter is applicable only when the mode is set to 'polling', and it can be applied only when you use this extension with WSO2 Stream Processor.
STRINGYesNo
table.name
The name of the table that needs to be monitored for data changes.
STRINGNoNo
polling.column
The column name  that is polled to capture the change data. It is recommended to have a TIMESTAMP field as the 'polling.column' in order to capture the inserts and updates.
Numeric auto-incremental fields and char fields can also be used as 'polling.column'. However, note that fields of these types only support insert change capturing, and the possibility of using a char field also depends on how the data is input.
**It is required to enter a value for this parameter only when the mode is 'polling'.**
STRINGYesNo
polling.interval
The time interval (specified in seconds) to poll the given table for changes.
This parameter is applicable only when the mode is set to 'polling'.
1INTYesNo
operation
The change event operation you want to carry out. Possible values are 'insert', 'update' or 'delete'. This parameter is not case sensitive. **It is required to specify a value only when the mode is 'listening'.**
STRINGNoNo
connector.properties
Here, you can specify Debezium connector properties as a comma-separated string. 
The properties specified here are given more priority over the parameters. This parameter is applicable only for the 'listening' mode.
Empty_StringSTRINGYesNo
database.server.id
An ID to be used when joining MySQL database cluster to read the bin log. This should be a unique integer between 1 to 2^32. This parameter is applicable only when the mode is 'listening'.
Random integer between 5400 and 6400STRINGYesNo
database.server.name
A logical name that identifies and provides a namespace for the database server. This parameter is applicable only when the mode is 'listening'.
{host}_{port}STRINGYesNo
wait.on.missed.record
Indicates whether the process needs to wait on missing/out-of-order records. 
When this flag is set to 'true' the process will be held once it identifies a missing record. The missing recrod is identified by the sequence of the polling.column value. This can be used only with number fields and not recommended to use with time values as it will not be sequential.
This should be enabled ONLY where the records can be written out-of-order, (eg. concurrent writers) as this degrades the performance.
falseBOOLYesNo
missed.record.waiting.timeout
The timeout (specified in seconds) to retry for missing/out-of-order record. This should be used along with the wait.on.missed.record parameter. If the parameter is not set, the process will indefinitely wait for the missing record.
-1INTYesNo

+
+Examples
+EXAMPLE 1
+```
+@source(type = 'cdc' , url = 'jdbc:mysql://localhost:3306/SimpleDB', 
+username = 'cdcuser', password = 'pswd4cdc', 
+table.name = 'students', operation = 'insert', 
+@map(type='keyvalue', @attributes(id = 'id', name = 'name')))
+define stream inputStream (id string, name string);
+```
+

+
In this example, the CDC source listens to the row insertions that are made in the 'students' table with the column name, and the ID. This table belongs to the 'SimpleDB' MySQL database that can be accessed via the given URL.

+

+EXAMPLE 2
+```
+@source(type = 'cdc' , url = 'jdbc:mysql://localhost:3306/SimpleDB', 
+username = 'cdcuser', password = 'pswd4cdc', 
+table.name = 'students', operation = 'update', 
+@map(type='keyvalue', @attributes(id = 'id', name = 'name', 
+before_id = 'before_id', before_name = 'before_name')))
+define stream inputStream (before_id string, id string, 
+before_name string , name string);
+```
+

+
In this example, the CDC source listens to the row updates that are made in the 'students' table. This table belongs to the 'SimpleDB' MySQL database that can be accessed via the given URL.

+

+EXAMPLE 3
+```
+@source(type = 'cdc' , url = 'jdbc:mysql://localhost:3306/SimpleDB', 
+username = 'cdcuser', password = 'pswd4cdc', 
+table.name = 'students', operation = 'delete', 
+@map(type='keyvalue', @attributes(before_id = 'before_id', before_name = 'before_name')))
+define stream inputStream (before_id string, before_name string);
+```
+

+
In this example, the CDC source listens to the row deletions made in the 'students' table. This table belongs to the 'SimpleDB' database that can be accessed via the given URL.

+

+EXAMPLE 4
+```
+@source(type = 'cdc', mode='polling', polling.column = 'id', 
+jdbc.driver.name = 'com.mysql.jdbc.Driver', url = 'jdbc:mysql://localhost:3306/SimpleDB', 
+username = 'cdcuser', password = 'pswd4cdc', 
+table.name = 'students', 
+@map(type='keyvalue'), @attributes(id = 'id', name = 'name'))
+define stream inputStream (id int, name string);
+```
+

+
In this example, the CDC source polls the 'students' table for inserts. 'id' that is specified as the polling colum' is an auto incremental field. The connection to the database is made via the URL, username, password, and the JDBC driver name.

+

+EXAMPLE 5
+```
+@source(type = 'cdc', mode='polling', polling.column = 'id', datasource.name = 'SimpleDB',
+table.name = 'students', 
+@map(type='keyvalue'), @attributes(id = 'id', name = 'name'))
+define stream inputStream (id int, name string);
+```
+

+
In this example, the CDC source polls the 'students' table for inserts. The given polling column is a char column with the 'S001, S002, ... .' pattern. The connection to the database is made via a data source named 'SimpleDB'. Note that the 'datasource.name' parameter works only with the Stream Processor.

+

+EXAMPLE 6
+```
+@source(type = 'cdc', mode='polling', polling.column = 'last_updated', datasource.name = 'SimpleDB',
+table.name = 'students', 
+@map(type='keyvalue'))
+define stream inputStream (name string);
+```
+

+
In this example, the CDC source polls the 'students' table for inserts and updates. The polling column is a timestamp field.

+

+EXAMPLE 7
+```
+@source(type='cdc', jdbc.driver.name='com.mysql.jdbc.Driver', url='jdbc:mysql://localhost:3306/SimpleDB', username='cdcuser', password='pswd4cdc', table.name='students', mode='polling', polling.column='id', operation='insert', wait.on.missed.record='true', missed.record.waiting.timeout='10',
+@map(type='keyvalue'), 
+@attributes(batch_no='batch_no', item='item', qty='qty'))
+define stream inputStream (id int, name string);
+```
+

+
In this example, the CDC source polls the 'students' table for inserts. The polling column is a numeric field. This source expects the records in the database to be written concurrently/out-of-order so it waits if it encounters a missing record. If the record doesn't appear within 10 seconds it resumes the process.

+

+EXAMPLE 8
+```
+@source(type = 'cdc', url = 'jdbc:oracle:thin://localhost:1521/ORCLCDB', username='c##xstrm', password='xs', table.name='DEBEZIUM.sweetproductiontable', operation = 'insert', connector.properties='oracle.outserver.name=DBZXOUT,oracle.pdb=ORCLPDB1' @map(type = 'keyvalue'))
+define stream insertSweetProductionStream (ID int, NAME string, WEIGHT int);
+
+```
+

+
In this example, the CDC source connect to an Oracle database and listens for insert queries of sweetproduction table

+

diff --git a/docs/api/latest.md b/docs/api/latest.md
index 97a4b12..d67ded9 100644
--- a/docs/api/latest.md
+++ b/docs/api/latest.md
@@ -1,4 +1,4 @@
-# API Docs - v2.0.5
+# API Docs - v2.0.6
 
 !!! Info "Tested Siddhi Core version: *5.1.7*"
     It could also support other Siddhi Core minor versions.
diff --git a/docs/index.md b/docs/index.md
index b3a2a06..04322f7 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -19,11 +19,11 @@ For information on Siddhi and i
 
 ## Latest API Docs 
 
-Latest API Docs is 2.0.5.
+Latest API Docs is 2.0.6.
 
 ## Features
 
-* cdc *(Source)*
 
The CDC source receives events when change events (i.e., INSERT, UPDATE, DELETE) are triggered for a database table. Events are received in the 'key-value' format.

There are two modes you could perform CDC: Listening mode and Polling mode.

In polling mode, the datasource is periodically polled for capturing the changes. The polling period can be configured.
In polling mode, you can only capture INSERT and UPDATE changes.

On listening mode, the Source will keep listening to the Change Log of the database and notify in case a change has taken place. Here, you are immediately notified about the change, compared to polling mode.

The key values of the map of a CDC change event are as follows.

For 'listening' mode: 
    For insert: Keys are specified as columns of the table.
    For delete: Keys are followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.
    For update: Keys are followed followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.

For 'polling' mode: Keys are specified as the columns of the table.#### Preparations required for working with Oracle Databases in listening mode

Using the extension in Windows, Mac OSX and AIX are pretty straight forward inorder to achieve the required behaviour please follow the steps given below

  - Download the compatible version of oracle instantclient for the database version from [here](https://www.oracle.com/database/technologies/instant-client/downloads.html) and extract
  - Extract and set the environment variable LD_LIBRARY_PATH to the location of instantclient which was exstracted as shown below
  
+* cdc *(Source)*
 
The CDC source receives events when change events (i.e., INSERT, UPDATE, DELETE) are triggered for a database table. Events are received in the 'key-value' format.

There are two modes you could perform CDC: Listening mode and Polling mode.

In polling mode, the datasource is periodically polled for capturing the changes. The polling period can be configured.
In polling mode, you can only capture INSERT and UPDATE changes.

On listening mode, the Source will keep listening to the Change Log of the database and notify in case a change has taken place. Here, you are immediately notified about the change, compared to polling mode.

The key values of the map of a CDC change event are as follows.

For 'listening' mode: 
    For insert: Keys are specified as columns of the table.
    For delete: Keys are followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.
    For update: Keys are followed followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.

For 'polling' mode: Keys are specified as the columns of the table.#### Preparations required for working with Oracle Databases in listening mode

Using the extension in Windows, Mac OSX and AIX are pretty straight forward inorder to achieve the required behaviour please follow the steps given below

  - Download the compatible version of oracle instantclient for the database version from [here](https://www.oracle.com/database/technologies/instant-client/downloads.html) and extract
  - Extract and set the environment variable LD_LIBRARY_PATH to the location of instantclient which was exstracted as shown below
  
     export LD_LIBRARY_PATH=<path to the instant client location>
   

  - Inside the instantclient folder which was download there are two jars xstreams.jar and ojdbc<version>.jar convert them to OSGi bundles using the tools which were provided in the <distribution>/bin for converting the ojdbc.jar use the tool spi-provider.sh|bat and for the conversion of xstreams.jar use the jni-provider.sh as shown below(Note: this way of converting Xstreams jar is applicable only for Linux environments for other OSs this step is not required and converting it through the jartobundle.sh tool is enough)
  
     ./jni-provider.sh <input-jar> <destination> <comma seperated native library names>
diff --git a/mkdocs.yml b/mkdocs.yml
index 1327b10..4887cda 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -34,6 +34,7 @@ pages:
 - Information: index.md
 - API Docs:
   - latest: api/latest.md
+  - 2.0.6: api/2.0.6.md
   - 2.0.5: api/2.0.5.md
   - 2.0.4: api/2.0.4.md
   - 2.0.3: api/2.0.3.md