Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [Module Name] CDCSOURCE task was submitted successfully, but the data could not be synchronized. #3331

Closed
2 of 3 tasks
cuanxiaopo opened this issue Mar 26, 2024 · 8 comments
Assignees
Labels
FAQ Frequently Asked Questions Invalid Invalid

Comments

@cuanxiaopo
Copy link

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

dink1.0.1, flink1.17。在测试整库同步时,任务提交成功,但是数据没有同步过去,全量数据,增量数据都无法同步过去,也没有报错。
EXECUTE CDCSOURCE cdc_sms WITH (
'connector' = 'mysql-cdc',
'hostname' = 'xxxxx',
'port' = '8527',
'username' = 'root',
'password' = '123456',
'checkpoint' = '3000',
'scan.startup.mode' = 'initial',
'parallelism' = '1',
'table-name' = 'sms.sms_type,sms.sms_receive',
'source.server-time-zone' = '+08:00',
'sink.connector' = 'jdbc',
'sink.url' = 'jdbc:mysql://xxxxxx:3306/sms?characterEncoding=utf-8&useSSL=false',
'sink.username' = 'root',
'sink.password' = '123456',
'sink.sink.db' = 'sms',
'sink.table.prefix' = 'ods_',
'sink.table.lower' = 'true',
'sink.table-name' = '#{tableName}',
'sink.driver' = 'com.mysql.jdbc.Driver',
'sink.sink.buffer-flush.interval' = '2s',
'sink.sink.buffer-flush.max-rows' = '100',
'sink.sink.max-retries' = '5',
'sink.auto.create' = 'true'
)

What you expected to happen

全量数据,增量数据正常同步,且实现表结构同步变更

How to reproduce

1

Anything else

No response

Version

1.0.0

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@cuanxiaopo cuanxiaopo added Bug Something isn't working Waiting for reply Waiting for reply labels Mar 26, 2024
Copy link

Hello @cuanxiaopo, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.

你好 @cuanxiaopo, 这个 issue 是关于 CDC/CDCSOURCE 的,所以我把它分配给了 @aiwenmo。如有任何问题,可以评论回复。

@github-actions github-actions bot changed the title [Bug] [Module Name] CDCSOURCE 任务提交成功,但是数据无法同步 [Bug] [Module Name] CDCSOURCE task was submitted successfully, but the data could not be synchronized. Mar 26, 2024
@cuanxiaopo
Copy link
Author

msyql到mysql的整库同步。

@cuanxiaopo
Copy link
Author

我之前测试0.7.5也遇到过类似的问题,现在用1.0.1之后还是同样的问题。表可以创建成功,数据无法同步过来

@cuanxiaopo
Copy link
Author

全量数据可以同步,增量数据无法同步

@Zzm0809 Zzm0809 added Invalid Invalid FAQ Frequently Asked Questions and removed Bug Something isn't working Waiting for reply Waiting for reply labels Mar 26, 2024
@Zzm0809
Copy link
Contributor

Zzm0809 commented Mar 26, 2024

set checkpoint 参数

@cuanxiaopo
Copy link
Author

set checkpoint 参数

全量同步语法中不是已经有 checkpoint设置了吗。在set下还是不行(SET 'execution.checkpointing.interval' = '3s';),增量数据仍然无法同步。

@Zzm0809
Copy link
Contributor

Zzm0809 commented Mar 26, 2024

set checkpoint 参数

全量同步语法中不是已经有 checkpoint设置了吗。在set下还是不行(SET 'execution.checkpointing.interval' = '3s';),增量数据仍然无法同步。

无法复现

@cuanxiaopo
Copy link
Author

set checkpoint 参数

全量同步语法中不是已经有 checkpoint设置了吗。在set下还是不行(SET 'execution.checkpointing.interval' = '3s';),增量数据仍然无法同步。

无法复现

1 mysql-mysql整库同步成功了,增量数据也可以了。(但是不能实现 表结构变更 的同步吗?)
2 昨天了解到某个测试库是polardb,并不是mysql。目前polardb-mysql单表可以,整库同步仍存在 这个问题。

@Zzm0809 Zzm0809 closed this as completed Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FAQ Frequently Asked Questions Invalid Invalid
Projects
None yet
Development

No branches or pull requests

3 participants