gray publish model refactor #12824

shiyiyue1102 · 2024-11-05T10:09:00Z

What is the purpose of the change

Gray Model Refactor
1.create config_info_gray model support gray publish and keep extentionable.
2. use new gray model to uniform beta(tag) model
3. auto migrate beta(tag) table data into config_info_gray during version upgrade period
4. modify beta(tag)publish logic, double write into beta(tag) and gray model on compatibity period ,support version rollback
5. modify config query logic , check beta (tag) from new gray models.
6. double write compatibity will still existed during further serval versions ,and then all beta ,tag code could be deleted on 3.0 version.

Gray History Supprt

Basic on gray model refactor , @MatthewAden support gray publish history and add gray model consistency between server and database. https://github.com/alibaba/nacos/issues/12217，support changed gray config and deleted gray configs .

Remove Aggr Code

Aggr model is used Within Alibaba Group ,this function is never supported on open source ,and also highly limited in Alibaba Group,
thus The aggr code is deleted in this pr for reducing code complexity and Improving the maintainability of the code repository.

Brief changelog

sql schema change :
new table config_info_gray model for new gray model:
CREATE TABLE config_info_gray (
id bigint unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
data_id varchar(255) NOT NULL COMMENT 'data_id',
group_id varchar(128) NOT NULL COMMENT 'group_id',
content longtext NOT NULL COMMENT 'content',
md5 varchar(32) DEFAULT NULL COMMENT 'md5',
src_user text COMMENT 'src_user',
src_ip varchar(100) DEFAULT NULL COMMENT 'src_ip',
gmt_create datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'gmt_create',
gmt_modified datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'gmt_modified',
app_name varchar(128) DEFAULT NULL COMMENT 'app_name',
tenant_id varchar(128) DEFAULT '' COMMENT 'tenant_id',
gray_name varchar(128) NOT NULL COMMENT 'gray_name',
gray_rule text NOT NULL COMMENT 'gray_rule',
encrypted_data_key varchar(256) NOT NULL DEFAULT '' COMMENT 'encrypted_data_key',
PRIMARY KEY (id),
UNIQUE KEY uk_configinfogray_datagrouptenantgray (data_id,group_id,tenant_id,gray_name),
KEY idx_dataid_gmt_modified (data_id,gmt_modified),
KEY idx_gmt_modified (gmt_modified)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT='config_info_gray'

table his_config_info change for gray publish history:
ALTER TABLE his_config_info ADD COLUMN publish_type varchar(50) DEFAULT 'formal' COMMENT 'publish type gray or formal'
ALTER TABLE his_config_info ADD COLUMN ext_info longtext DEFAULT NULL COMMENT 'ext info',

* 34567 * refactor gray model * pmd checkstyle * 兼容性处理 * 灰度增量对账 * 升级内容字段编码，添加gray表 * pmd and checkstyle * merge develop fix conflict * derby create gray table * modify gray table sql * derby sql bugfix * fix test case * fix test case * fix test case

…libaba#12507) * Refine the formal configuration and add additional information（alibaba#12217） * Change the "configType" field to "publishType"（alibaba#12217） * Improve the gray configuration（alibaba#12217） * add the reconciliation mechanism of gray configuration（alibaba#12217） * Change the JSON format of gray rule.（alibaba#12217） * Remove redundant queries（alibaba#12217） * fix reconciliation mechanism of gray configuration.（alibaba#12217） * Adjust the way of obtaining grayName.（alibaba#12217） * Change the variable name.（alibaba#12217）

…h-refactor # Conflicts: # config/src/main/java/com/alibaba/nacos/config/server/constant/PropertiesConstant.java # config/src/main/java/com/alibaba/nacos/config/server/remote/ConfigChangeClusterSyncRequestHandler.java # config/src/main/java/com/alibaba/nacos/config/server/service/dump/disk/ConfigRawDiskService.java # config/src/test/java/com/alibaba/nacos/config/server/service/dump/disk/ConfigRawDiskServiceTest.java # console/src/main/resources/static/index.html # console/src/main/resources/static/js/main.js

github-actions · 2024-11-05T10:09:11Z

Thanks for your this PR. 🙏
Please check again for your PR changes whether contains any usage/api/configuration change such as Add new API , Add new configuration, Change default value of configuration.
If so, please add or update documents(markdown type) in docs/next/ for repository nacos-group/nacos-group.github.io

感谢您提交的PR。 🙏
请再次查看您的PR内容，确认是否包含任何使用方式/API/配置参数的变更，如：新增API、新增配置参数、修改默认配置等操作。
如果是，请确保在提交之前，在仓库nacos-group/nacos-group.github.io中的docs/next/目录下添加或更新文档（markdown格式）。

wuyfee · 2024-11-05T10:53:18Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

wuyfee · 2024-11-05T11:01:48Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

wuyfee · 2024-11-07T15:53:42Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

wuyfee · 2024-11-07T15:58:09Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

wuyfee · 2024-11-11T02:30:18Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

wuyfee · 2024-11-11T03:24:38Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

codecov-commenter · 2024-11-11T06:34:43Z

Codecov Report

Attention: Patch coverage is 70.45305% with 450 lines in your changes missing coverage. Please review.

Project coverage is 72.15%. Comparing base (e2d44f2) to head (23183ec).
Report is 2 commits behind head on develop.

Files with missing lines	Patch %	Lines
.../server/service/ConfigGrayModelMigrateService.java	0.00%	84 Missing ⚠️
.../config/server/service/ConfigOperationService.java	67.82%	26 Missing and 11 partials ⚠️
...rnal/ExternalConfigInfoGrayPersistServiceImpl.java	79.44%	15 Missing and 22 partials ⚠️
...dded/EmbeddedConfigInfoGrayPersistServiceImpl.java	81.81%	11 Missing and 21 partials ⚠️
...acos/config/server/service/ConfigCacheService.java	69.23%	14 Missing and 14 partials ⚠️
...erver/service/dump/DumpChangeGrayConfigWorker.java	63.38%	19 Missing and 7 partials ⚠️
...er/service/dump/disk/ConfigRocksDbDiskService.java	23.07%	18 Missing and 2 partials ⚠️
...plugin/datasource/mapper/ConfigInfoGrayMapper.java	4.76%	20 Missing ⚠️
...a/nacos/config/server/utils/ConfigExtInfoUtil.java	77.77%	11 Missing and 7 partials ⚠️
...aba/nacos/config/server/model/ConfigCacheGray.java	45.45%	11 Missing and 1 partial ⚠️
... and 33 more

Additional details and impacted files

@@              Coverage Diff              @@
##             develop   #12824      +/-   ##
=============================================
- Coverage      72.63%   72.15%   -0.49%     
+ Complexity      9802     9783      -19     
=============================================
  Files           1278     1283       +5     
  Lines          41015    41349     +334     
  Branches        4330     4366      +36     
=============================================
+ Hits           29792    29836      +44     
- Misses          9159     9411     +252     
- Partials        2064     2102      +38

Files with missing lines	Coverage Δ
...n/java/com/alibaba/nacos/api/common/Constants.java	`58.33% <ø> (ø)`
...java/com/alibaba/nacos/api/model/v2/ErrorCode.java	`100.00% <100.00%> (ø)`
...libaba/nacos/config/server/constant/Constants.java	`76.92% <ø> (ø)`
...cos/config/server/constant/PropertiesConstant.java	`0.00% <ø> (ø)`
...a/nacos/config/server/model/ConfigHistoryInfo.java	`85.48% <100.00%> (+1.55%)`	⬆️
...nfig/server/model/event/ConfigDataChangeEvent.java	`81.81% <100.00%> (+22.35%)`	⬆️
...cos/config/server/model/event/ConfigDumpEvent.java	`94.00% <100.00%> (+0.81%)`	⬆️
...onfig/server/model/event/LocalDataChangeEvent.java	`100.00% <ø> (+25.00%)`	⬆️
.../remote/ConfigChangeBatchListenRequestHandler.java	`85.71% <100.00%> (ø)`
.../config/server/remote/RpcConfigChangeNotifier.java	`90.58% <100.00%> (+1.57%)`	⬆️
... and 61 more

... and 12 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2d44f2...23183ec. Read the comment docs.

wuyfee · 2024-11-11T07:02:09Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

wuyfee · 2024-11-11T08:36:24Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

wuyfee · 2024-11-11T09:50:29Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

wuyfee · 2024-11-11T10:09:41Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

2. gray model upgrade compatible 3. delete aggr model code

…to develop-graypublish-refactor

shiyiyue1102 · 2024-11-11T10:49:00Z

create a new pr ,close current

wuyfee · 2024-11-11T10:56:45Z

$\color{red}{FAILURE}$
DETAILS
✅ - docker: success
❌ - deploy (standalone & cluster & standalone_auth): failure
❌ - e2e-java-test (standalone & cluster & standalone_auth): skipped
❌ - e2e-go-test (standalone & cluster): skipped
❌ - e2e-cpp-test (standalone & cluster): skipped
❌ - e2e-csharp-test (standalone & cluster): skipped
❌ - e2e-nodejs-test (standalone & cluster): skipped
❌ - e2e-python-test (standalone & cluster): skipped
✅ - clean (standalone & cluster & standalone_auth): success

shiyiyue1102 and others added 15 commits July 26, 2024 17:04

删除aggr聚合配置代码

b4fc9f4

beta tag持久化层代码设置为废弃

55af10a

修复beta tag优先级问题，解决加密key npe

6cde885

bugfix

9e486a4

简化水平通知模型参数

62b4083

优化灰度升级期间的兼容性逻辑

b7db53f

修改derby初始化sql

1b2a5c6

test case fix

b37bb56

app labels client version fix

e59e204

删除config_info_aggr , 修改sql文件

8d37ceb

checkstyle fix

a4339d6

test case fix

2624497

shiyiyue1102 added 2 commits November 5, 2024 18:12

test case and check style fix

8bb16bb

test case fix

6e0a9e2

shiyiyue1102 added 7 commits November 6, 2024 17:46

又该beta，tag的默认优先级顺序，修复旧版本删除beta时候同步bug

e6b7b64

简化灰度水平通知逻辑，减少重复事件。checkstyle

144900c

checkstyle，pmd，升级过程中新旧版本兼容问题处理

3cb2c7d

修改默认配置中sql链接串的时区，和dumpchange一致性相关

f68d4dc

PRIORITY拼写错误

8253d81

testcase fix

0a63794

testcase fix

ad318c0

fix test case

d311208

fix test case

5384c10

fix test case

acec03b

simplify gray model

53d8fb9

shiyiyue1102 added 4 commits November 11, 2024 16:37

ext info coverage and rename method

9bd8661

checkstyle fix

c1b9c9a

rat fix

85e06c9

delete ConfigAllInfo4Gray

a52b504

shiyiyue1102 added 2 commits November 11, 2024 18:15

1. simplify gray model code

fba3702

2. gray model upgrade compatible 3. delete aggr model code

Merge remote-tracking branch 'origin/develop-graypublish-refactor' in…

23183ec

…to develop-graypublish-refactor

shiyiyue1102 closed this Nov 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gray publish model refactor #12824

gray publish model refactor #12824

shiyiyue1102 commented Nov 5, 2024 •

edited

Loading

github-actions bot commented Nov 5, 2024

wuyfee commented Nov 5, 2024

wuyfee commented Nov 5, 2024

wuyfee commented Nov 7, 2024

wuyfee commented Nov 7, 2024

wuyfee commented Nov 11, 2024

wuyfee commented Nov 11, 2024

codecov-commenter commented Nov 11, 2024 •

edited

Loading

wuyfee commented Nov 11, 2024

wuyfee commented Nov 11, 2024

wuyfee commented Nov 11, 2024

wuyfee commented Nov 11, 2024

shiyiyue1102 commented Nov 11, 2024

wuyfee commented Nov 11, 2024

gray publish model refactor #12824

gray publish model refactor #12824

Conversation

shiyiyue1102 commented Nov 5, 2024 • edited Loading

What is the purpose of the change

Brief changelog

github-actions bot commented Nov 5, 2024

wuyfee commented Nov 5, 2024

wuyfee commented Nov 5, 2024

wuyfee commented Nov 7, 2024

wuyfee commented Nov 7, 2024

wuyfee commented Nov 11, 2024

wuyfee commented Nov 11, 2024

codecov-commenter commented Nov 11, 2024 • edited Loading

Codecov Report

wuyfee commented Nov 11, 2024

wuyfee commented Nov 11, 2024

wuyfee commented Nov 11, 2024

wuyfee commented Nov 11, 2024

shiyiyue1102 commented Nov 11, 2024

wuyfee commented Nov 11, 2024

shiyiyue1102 commented Nov 5, 2024 •

edited

Loading

codecov-commenter commented Nov 11, 2024 •

edited

Loading